modified check_validity.py and added sample dataset to test functionality

src/backend/evaluate_model.py

@@ -69,7 +69,8 @@ class Evaluator:
             dict: A dictionary containing evaluation results.
         """
         try:
-            df = pd.read_csv(envs.SOURCE_PATH)
+            # df = pd.read_csv(envs.SOURCE_PATH)
+            df = pd.read_csv(envs.SAMPLE_DATASET_PATH)
             generated_summaries_df = self.summary_generator.generate_summaries(df)
 
             avg_summary_len = self.summary_generator.avg_length
@@ -88,7 +89,7 @@ class Evaluator:
 
             return results
         except FileNotFoundError:
-            logging.error(f"File not found: {envs.SOURCE_PATH}")
+            logging.error(f"File not found: {envs.SAMPLE_DATASET_PATH}")
             raise
         except Exception as e:
             logging.error(f"Error during evaluation: {e}")

src/datasets/sample_dataset.csv

@@ -0,0 +1,11 @@
+text,dataset
+"Paul Merson has restarted his row with Andros Townsend after the Tottenham midfielder was brought on with only seven minutes remaining in his team's 0-0 draw with Burnley on Sunday. 'Just been watching the game, did you miss the coach? #RubberDub #7minutes,' Merson put on Twitter. Merson initially angered Townsend for writing in his Sky Sports column that 'if Andros Townsend can get in (the England team) then it opens it up to anybody.' Paul Merson had another dig at Andros Townsend after his appearance for Tottenham against Burnley . Townsend was brought on in the 83rd minute for Tottenham as they drew 0-0 against Burnley . Andros Townsend scores England's equaliser in their 1-1 friendly draw with Italy in Turin on Tuesday night . The former Arsenal man was proven wrong when Townsend hit a stunning equaliser for England against Italy and he duly admitted his mistake. 'It's not as though I was watching hoping he wouldn't score for England, I'm genuinely pleased for him and fair play to him – it was a great goal,' Merson said. 'It's just a matter of opinion, and my opinion was that he got pulled off after half an hour at Manchester United in front of Roy Hodgson, so he shouldn't have been in the squad. 'When I'm wrong, I hold my hands up. I don't have a problem with doing that - I'll always be the first to admit when I'm wrong.' Townsend hit back at Merson on Twitter after scoring for England against Italy . Sky Sports pundit  Merson (centre) criticised Townsend's call-up to the England squad last week . Townsend hit back at Merson after netting for England in Turin on Wednesday, saying 'Not bad for a player that should be 'nowhere near the squad' ay @PaulMerse?' Any bad feeling between the pair seemed to have passed but Merson was unable to resist having another dig at Townsend after Tottenham drew at Turf Moor.",summeval_valid
+"Chelsea have made an offer for FC Tokyo's 22-year-old forward Yoshinori Muto, according to club president Naoki Ogane. The Japan international, who has played for the J-League side since 2013, will join Chelsea's Dutch partner club Vitesse Arnhem on loan next season if he completes a move to Stamford Bridge this summer. Ogane claims that Chelsea's interest in Muto is not connected to the £200million sponsorship deal they signed with Japanese company Yokohama Rubber in February. FC Tokyo forward Yoshinori Muto (centre) brings the ball forward against Albirex Niigata in March . FC Tokyo president Naoki Ogane claims that Chelsea have made a bid for Japan international Muto . Muto tussles with Yuji Nakazawa of Yokohama F.Marinos during a J-League clash last month . Age: 22 . Club: FC Tokyo . Appearances: 37 . Goals: 16 . International caps (Japan): 11 . International goals: 1 . Did you know? Muto graduated from Keio University in Tokyo with an economics degree two weeks ago . Speaking to Sports Nippon, Ogane said: 'It is true that Chelsea sent us an offer for Muto. 'It is a formal offer with conditions. They want to acquire him in the summer.' Muto, who only graduated from Keio University a fortnight ago after completing an economics degree, would be the first Japanese player to represent Chelsea if he moves to west London. He has earned 11 caps for his country after signing his first professional contract in 2014, scoring once for the Samurai Blue. A £4million deal for the youngster has been mooted, but Muto admits that he isn't sure if he will join the Premier League title chasers despite being pleased with their bid. He said: 'I have not decided yet at all. It is an honour for me to receive this offer from a great club.' Muto scored 13 times in his debut season with FC Tokyo and was named in the J-League's best XI. Muto admits it is an 'honour' to receive an offer from Chelsea although he has not yet decided to join the club . Muto, pictured in action against Ventforet Kofu has scored three goals in four games so far this season . The 22-year-old has a shot at goal during Japan's Asian Cup match against Palestine in January this year . He has continued his fine form during the current campaign, helping his club to third place in the division with three goals in four games. Yokohama Rubber - one of the world's largest tyre manufacturers - will become Chelsea's official shirt sponsors from the start of the 2015-16 season. The initial five-year deal is the biggest in the club's history, with the Blues now considering a two-week pre-season tour of Japan this summer.",summeval_valid
+"Babies given antibiotics in the first six months of life are more likely to be fat as toddlers, a large-scale study has found. The researchers said say that just as antibiotics are used to make farm animals put on weight, the may also be fattening our children. Writing in the respected medical journal Pediatrics, they said that the widely-prescribed drugs could be contributing to the obesity epidemic. A third of 10-11 year olds and more than a fifth of 4-5 year olds in England are overweight or obese, leading to fears that today's generation will be the first to die at an earlier age than their parents. Obesity: Babies given antibiotics in the first six months of life are more likely to be fat as toddlers, a large-scale study has found (file photo) The Finnish researchers compared the weight and height of more than 12,000 healthy two year old with records on antibiotic prescription. By two years-old, one in five boys and one in ten girls was overweight or obese. And children who had taken antibiotics as young babies were particularly likely to be overweight. Repeated prescriptions before the age of two also raised the odds of being a fat toddler. Boys seemed particularly prone weight gain after being given antibiotics. They were also slightly taller than boys who hadn't been given the drugs. The study didn't prove that antibiotics were causing weight gain. But if they do, it may be because they kill of bugs in the gut that would normally feed on some of the food eaten. This frees up more food for the body. Killing certain gut bugs may also increase appetite. Lead author Dr Antti Saari, of Kuopio University Hospital, warned: 'Antibiotic exposure before six months of age, or repeatedly during infancy, was associated with increased body mass in healthy children. 'Such effects may play a role in the worldwide childhood obesity epidemic and highlight the importance of judicious use of antibiotics in infancy. The worldwide obesity epidemic is real, and is more pronounced for boys. 'Epidemic': By two years-old, one in five boys and one in ten girls was overweight or obese and children who had taken antibiotics as young babies were particularly likely to be overweight (file photo) 'An increase in the use of antibiotics could be an additional contributing factor to the development of excess weight problems. 'The crucial role of antibiotics in the improvement of human health is unquestionable but their extended use today has undesirable and unexpected consequences.' Previous research has found that babies given antibiotics are at higher risk of eczema and digestive problems. The studies come amid growing concern that the over-prescription of antibiotics is leading to the pills losing their power and making common infections harder to treat. The Government has warned that a new superbug could infect up to 200,000 Britons and kill 80,000 in a single outbreak.",summeval_valid
+"This is the embarrassing moment a Buckingham Palace guard slipped and fell on a manhole cover in front of hundreds of shocked tourists as he took up position in his sentry box. The Queen's Guard was left red-faced after he slipped on a manhole cover during the popular Changing of the Guard - and unfortunately for him the entire incident was caught on camera. He lost his footing and slid sideways, knocking his bearskin on the side of the box and dropping his rifle. The Queen's Guard (pictured) slipped on a manhole cover during the popular Changing of the Guard at Buckingham Palace last week. Unfortunately for him, the entire incident was caught on a tourist's camera . The embarrassed soldier quickly scrambled to his feet as his colleagues marched past as if nothing had happened. But the young guard started to blush as he peered at the crowd from under his bearskin and realised how many people had seen his slapstick moment. Holidaymaker David Meadwell recorded the unscheduled manouevre outside Buckingham Palace on Thursday afternoon. Mr Meadwell, 50, from Newcastle-upon-Tyne, said: 'I was with my family for a trip to London and I thought I'd take some pictures of the changing of the guards. Tourist David Meadwell shot this footage of the Changing of the Guard last week when the incident unfolded . The moment it all started to go wrong: The guard leans heavily to the side as he appears to slip . The unidentified young guard's legs appear to get a bit tangled as he tries to turn to march away . The guard, wearing full regalia, falls heavily to the floor still clutching his rifle following the slip up . 'The first group changed successfully, without any problems, and so I decided to video the next group doing it. 'I didn't expect anything like this to happen - he went flying. There were quite a few people around and there were a lot of gasps as he went down. 'I think he just slipped on a manhole cover, he looked so embarrassed.' The unnamed solider is thought to have slipped because of the metal protectors nailed to the soles of his boots. Tourists gather in their thousands to watch the changing of the guard outside Buckingham Palace at 11.30am every day in the summer and every other day in the winter. The Guard comprises two detachments, one each for Buckingham Palace and St James's Palace, under the command of the Captain of The Queen's Guard. Contrary to popular belief they are not purely ceremonial and are fully operational soldiers. The Ministry of Defence said they would not comment on 'a young man falling over while doing his job'. The embarrassed guard hastily scrambled to his feet following the unfortunate tumble at the palace . The incident took place in front of hundreds of tourists who were watching the Changing of the Guard .",summeval_valid
+"(CNN)One of the biggest TV events of all time is being reimagined for new audiences. ""Roots,"" the epic miniseries about an African-American slave and his descendants, had a staggering audience of over 100 million viewers back in 1977. Now A&E networks are remaking the miniseries, to air in 2016. A&E, Lifetime and History (formerly the History Channel) announced Thursday that the three networks would simulcast a remake of the saga of Kunta Kinte, an African who was captured, shipped to America and sold into slavery to work on a Virginia plantation. LeVar Burton, who portrayed Kinte in the original, will co-executive produce the new miniseries. A press release describes the new version as ""original"" and ""contemporary"" and will draw more from Alex Haley's classic novel, ""Roots: The Saga of an American Family."" Producers will consult scholars in African and African-American history for added authenticity. ""We are proud to bring this saga to fans of the original, as well as to a new generation that will experience this powerful and poignant tale for the first time,"" said Dirk Hoogstra, History's executive vice president and general manager. ""Audiences will once again feel the impact of Kunta Kinte's indomitable spirit."" Executive producer Mark Wolper, son of the original's producer David L. Wolper, added, ""Kunta Kinte began telling his story over 200 years ago and that story went through his family lineage, to Alex Haley, to my father, and now the mantle rests with me. Like Kunta Kinte fought to tell his story over and over again, so must we."" The remade ""Roots"" will encounter a new generation of viewers who have witnessed Barack Obama make history as the nation's first African-American president and ""12 Years a Slave"" win the Oscar for Best Picture, but also widespread racial unrest over police treatment of black suspects in many U.S. cities. ""My career began with 'Roots' and I am proud to be a part of this new adaptation,"" said Burton. ""There is a huge audience of contemporary young Americans who do not know the story of 'Roots' or its importance.""",summeval_valid
+"Police are investigating claims by a former royal footman that palace aides tried to force him into an orgy, it was revealed yesterday. Christopher Lawler said he was pinned to a chair and groped by a male member of staff on his first day working at Clarence House. The ordeal left him in tears and he left the job the same day. He finally decided to contact palace officials again last year after inquiries began into alleged cover-ups of child sex abuse by a VIP paedophile ring in the 1970s and 80s. Police are investigating claims made by a former royal footman that Clarence House aides tried to force him into an orgy in the 1970s . But his complaints were ignored for months before the police were finally notified, he claims. Mr Lawler, now 64, said the attempted abuse at the Queen Mother’s London residence happened in January 1978 when he was 27. He said he was accosted after he walked into a bedroom to look for a pen on his first shift. Two men offered him a drink before two other men joined them – one a senior member of staff, he said. He was asked if he was gay before a younger man took his trousers off and began performing a sex act on himself. ‘That prompted another guy to put his hand on my leg and then he grabbed me,’ he told the Sunday People. ‘I was staggered. The younger man then came up behind me and gripped me, holding me in the chair. Mr Lawler worked at Clarence House when the Queen Mother used it as her London residence . ‘They were trying to undo my trousers but I managed to jump up and burst out of the room.’ Afterwards, Mr Lawler said he was followed by two men and threatened to keep quiet. A complaint he made that day allegedly drew an angry response from Clarence House – so he packed his bags immediately. Mr Lawler, a former Port of Liverpool police officer, said he was ‘haunted’ by the incident for years. After hearing about the probe into historical cases of child abuse last year he wrote to the Palace, but was twice rebuffed. Months later the complaint was finally passed to Scotland Yard. A retired royal aide, who is now in his 80s, was reportedly interviewed but denied he was involved because he was working for the Queen in Balmoral at the time. Mr Lawler has now been told by the Palace that the Royal Household would work ‘exhaustively and co-operatively’ with any police probe. A police spokesman said it would be inappropriate to comment ‘as investigations continue’.",summeval_valid
+"An Oregon couple announced they are expecting a child in a rap video they made set to the theme  from '90s television sitcom 'The Fresh Prince of Bel-Air.' The clip, which features Jesse and Melissa Meek rapping as they drive in a car, has been viewed over 1.7 million times on YouTube. 'In Happy Valley, Oregon, livin' it wed, bought our first house and started to build our homestead,' Melissa says in the clip. Parents: Jesse and Melissa Meek announced they are expecting a child in a video set to the theme song from '90s television sitcom 'The Fresh Prince of Bel-Air' The original song for the popular NBC sitcom, sung by star Will Smith, details how his character grew up in West Philadelphia, where he got into a neighborhood fight - and at his mother's insistence, moved to his aunt and uncle's home in their wealthy Los Angeles neighborhood. In the Meeks' parody video, Melissa raps, 'When our family and friends, who were up to no good, started asking questions about parenthood. 'We told one little lie, said ""It's not time yet."" Parents said ""We're ready for grandkids, get a move on it!""' Jesse raps that the couple thought it would take longer than the two months the pair spent trying to conceive. Melissa says in the video 'I woke up in the morning about 7 or 8 and I thought to myself, ""Oh man, I'm late!"" Looked at the test and it was finally there. The little plus sign. We're now three. Not a pair.' At the end of the video, the Meeks smile and share a sonogram of their unborn child. It took five takes to film the clip, the couple told KPTV. After finding out the gender of the child, another video is a possibility for the Meeks, the couple told the Fox affiliate. Original: Will Smith is seen here rapping the theme for 'The Fresh Prince of Bel-Air' during the show's title sequence . Big reveal: At the end of the clip, the Meeks share a sonogram of their unborn child . According to KPTV, the video was made so loved ones living far away could know about the baby. Melissa told the Fox affliate of the video's success 'It was completely unexpected. Like, that was the last thing we thought would happen, that it would blow up to what it had, or what it has.' Jesse told the Oregonian 'It has been a lot of fun, but definitely way more than we ever expected.' He is the great-great-grandson of Oregon pioneer Joseph Lafayette Meek, the newspaper reported. The Oregonian reported that Melissa earlier made a video which captured Jesse's reaction when he found out about the pregnancy. Jesse learned the news after reading a tag Melissa placed on their dog, which indicated a baby was on the way. A Phoenixville, Pennsylvania, couple made a pregnancy announcement video using 'The Fresh Prince of Bel-Air' theme song last year and recreated the show's title sequence, People reported.",summeval_valid
+"A dress worn by Vivien Leigh when she played Scarlett O'Hara in the classic 1939 film Gone With the Wind has fetched $137,000 at auction. Heritage Auctions offered the gray jacket and skirt, featuring a black zigzag applique, plus more than 150 other items from the Academy Award-winning film at auction on Saturday in Beverly Hills, California. The dress - a jacket and full skirt ensemble - was worn in several key scenes in the 1939 movie, including when Scarlett O'Hara encounters Rhett Butler, played by Clark Gable, and when she gets attacked in the shanty town. Scroll down for video . An outfit worn in several scenes of the 1939 film Gone With The Wind by Vivien Leigh as she played Scarlett O'Hara sold for $137,000 at auction on Saturday . The dress - a jacket and full skirt ensemble - was worn in several key scenes in the 1939 movie but has suffered a little with age and has faded to light gray from original slate blue-gray color . The outfit has suffered a little with age, however. When Leigh wore it in the movie, it was slate blue-gray but over the years it has faded to light gray. It was one of more than 150 items that were part of the private collection of James Tumblin, formerly in charge of the hair and makeup department at Universal Studios. Tumblin began collecting onscreen costumes, props and behind-the-scenes artifacts from the film in the 1960s, amassing a collection of more than 300,000 pieces of memorabilia. During a visit to the Western Costume Company he spotted the Scarlett O'Hara dress on the floor. He learned that the dress was about to be thrown away and negotiated a deal to buy it for $20. Tumblin has 'devoted his life and efforts to promoting Hollywood and this film, touring his items throughout the United States,' said Kathleen Guzman, managing director of Heritage Auctions. Gone With The Wind, which celebrated its 75th anniversary last year, was based on Margaret Mitchell's 1936 best-selling book about a spoiled Old South socialite, played by Vivien Leigh, and co-starred Clark gable as Rhett Butler . Hattie McDaniel (left), Olivia DeHavilland (middle), and Vivien Leigh: McDaniel famously became the first African-American actor to be nominated for and win an Academy Award . Other top selling items from the auction were a straw hat worn by Leigh that sold for $52,500; the trousers and jacket from a suit worn by Clark Gable as Rhett Butler, selling for $55,000; and a black bonnet worn by both Leigh and Olivia De Havilland as Melanie Wilkes, which fetched $30,000. Gone With The Wind, which celebrated its 75th anniversary last year, was based on Margaret Mitchell's 1936 best-selling book about a spoiled Old South socialite. Actress Hattie McDaniel, who played Scarlett's devoted nanny Mammy, a slave, famously became the first African-American actor to be nominated for and win an Academy Award.",summeval_valid
+"A two-year-old boy is recovering after falling into a cheetah exhibit at the Cleveland Metroparks Zoo after his parents dangled him over the edge, officials said. The toddler's mother was holding him and another child when he slipped and fell between 10 to 12ft and into the pit on Saturday around 3pm. The child was rescued by his parents before firefighters and paramedics arrived on the scene. Scroll down for video . A mother was holding the two-year-old boy and another child when the toddler slipped and fell into the cheetah exhibit at the Cleveland Metroparks Zoo (file photo of cheetahs at the Cleveland zoo) The boy was rescued by his parents from the pit (pictured) before firefighters and paramedics arrived on the scene. He suffered from minor bumps and bruises and was listed in stable condition at the hospital . He is listed in stable condition after being taken to MetroHealth Medical Center and suffered from minor bruises and bumps. The boy's leg was injured in the fall, but he was not attacked by the animals, Dr. Christopher Kuhar, the zoo's executive director told Fox 8. Michael Lurie and his family were at the Cheetah exhibit when they heard the child scream. 'You saw how far the drop was and you just couldn't believe the kid didn't hurt himself from falling down on the ground,' Lurie told WKYC. 'I was just shocked,' he said. 'I didn't understand how the parents let the kid go over the thing.' The cheetahs did not approach the boy or his parents while in the pit, according to zoo officials. Zoo visitor Terra Lurie believes the boy was not approached by the fast feline because they were frightened. 'I think they were just curious as to what was going on and why somebody was in the pen with them,' she said. 'It's not every day that somebody is just in the pen with them.' 'And everyone else is screaming and they probably got scared.' Kuhar said the zoo had received 'a number of eyewitness accounts' that indicate the 'strong likelihood that the child was dangled over the railing,' he told NewsNet5. Cleveland Metroparks Zoo has plans to press child endangerment charges against the family on Monday. The exhibit was closed following the child's fall. Zoo visitor Michael Lurie was at the cheetah exhibit when he heard the child scream. He said he was 'shocked' and 'didn't understand how the parents let the kid' go over the railing and into the pit . Cleveland Metroparks Zoo plans to press child endangering charges against the child's mother (above file photo of visitors at the Cleveland zoo)",summeval_valid
+"The owners of this house better not leave too quickly, after a speed camera was angled directly on their front door. The bright yellow gatso had previously enforced the 30mph speed limit for motorists along the residential road in Handsworth, Birmingham. However, it has not been working for two years after every single fixed device was switched off in the West Midlands. Big Brother is watching: A speed camera has been turned round and is pointing at this house in Birmingham, West Midlands . The speed camera has not been working for more than two years . Around 300 speed and traffic camera, using old technology, were turned off across the region in March 2013 . In there place, speed enforcement operations have been carried out by a small number of mobile camera units, fixed cameras on motorways and traffic officers on patrol. Mystery surrounds who had re-pointed the camera, but a spokesman for Birmingham City Council said they were aware of it. One of their engineers will now be visiting the site and the camera could be removed completely. 'Fixed location safety cameras have been decommissioned across the West Midlands since 2013 as the technology inside them had become obsolete,' the spokesman said. 'Plans for a pilot at a limited number of sites, using digital technology, is currently in development. 'Now the issue with this camera in Wellington Road has been brought to our attention, we will take any appropriate action at the site.' The spokesman confirmed that there were no plans to include the camera in Wellington Road in the new pilot. The owners of the house were not available for comment.",summeval_valid

src/envs.py

@@ -22,4 +22,5 @@ DEVICE = "cpu"
 API = HfApi(token=TOKEN)
 
 SOURCE_PATH = "src/datasets/leaderboard_dataset.csv"
+SAMPLE_DATASET_PATH = "src/datasets/sample_dataset.csv"
 HEM_PATH = 'vectara/hallucination_evaluation_model'

src/submission/check_validity.py

@@ -5,7 +5,7 @@ from collections import defaultdict
 import huggingface_hub
 from huggingface_hub import ModelCard
 from huggingface_hub.hf_api import ModelInfo
-from transformers import AutoConfig
+from transformers import AutoConfig, AutoTokenizer
 from transformers.models.auto.tokenization_auto import tokenizer_class_from_name, get_tokenizer_config
 
 def check_model_card(repo_id: str) -> tuple[bool, str]:
@@ -31,24 +31,19 @@ def check_model_card(repo_id: str) -> tuple[bool, str]:
 
 
 def is_model_on_hub(model_name: str, revision: str, token: str = None, trust_remote_code=False, test_tokenizer=False) -> tuple[bool, str]:
-    """Makes sure the model is on the hub, and uses a valid configuration (in the latest transformers version)"""
     try:
         config = AutoConfig.from_pretrained(model_name, revision=revision, trust_remote_code=trust_remote_code, token=token)
         if test_tokenizer:
-            tokenizer_config = get_tokenizer_config(model_name)
-            if tokenizer_config is not None:
-                tokenizer_class_candidate = tokenizer_config.get("tokenizer_class", None)
-            else:
-                tokenizer_class_candidate = config.tokenizer_class
-
-
-            tokenizer_class = tokenizer_class_from_name(tokenizer_class_candidate)
-            if tokenizer_class is None:
+            try:
+                tk = AutoTokenizer.from_pretrained(model_name, revision=revision, trust_remote_code=trust_remote_code, token=token)
+            except ValueError as e:
                 return (
                     False,
-                    f"uses {tokenizer_class_candidate}, which is not in a transformers release, therefore not supported at the moment.",
+                    f"uses a tokenizer which is not in a transformers release: {e}",
                     None
                 )
+            except Exception as e:
+                return (False, "'s tokenizer cannot be loaded. Is your tokenizer class in a stable transformers release, and correctly configured?", None)
         return True, None, config
 
     except ValueError: