Board index Archives College area archives
NAQT Security Discussion
Old college threads.
rhentzel
Contact rhentzel
Post by rhentzel » Wed Mar 20, 2013 3:01 pm
As mentioned at the time that the previous announcement of unauthorized question access was reported, NAQT has been continuing its security review, which turned up the three cases announced on March 20.
There is, of course, more work to be done, though NAQT is confident that the holes that led to these four cases were fixed as of May 2012 and have not compromised any tournaments from this competition year.
We are interested in integrating community input and suggestions into our review process, so I have started this thread to gather suggestions for preventing (and/or detecting) unauthorized access to questions. We assume the focus will be on NAQT's administrative website, but any of NAQT's policies are theoretically up for discussion.
If you know of, or think you know of, an exploitable security hole on NAQT's website, please e-mail us privately at [email protected] and give us one month to fix it. If it hasn't been fixed by then, we will consider ourselves to have forfeited the right to complain about its public announcement (though we will still ban anybody determined to have taken advantage of it).
R. Robert Hentzel, Chief Technical Officer of National Academic Quiz Tournaments
Re: NAQT Security Discussion
Post by Sen. Estes Kefauver (D-TN) » Wed Mar 20, 2013 3:09 pm
I would like an explanation of whether or not there is truth to the rumors that Sameer from Bellarmine accidentally accessed the Sectionals tournament before TI, and thus chose to sit out.
I frankly think that, at this point, NAQT should stop hiring high schoolers. The system is clearly flawed, lots of high school writers are unproven in an ethical sense and question writing sense, and Joe Brosch's case demonstrates that. I don't think trusting that some 16 year old will do the right thing is a tenable security position for NAQT to take when the bulk of their product is geared towards those very high schoolers.
Post by Important Bird Area » Wed Mar 20, 2013 3:19 pm
Horned Screamer wrote: Sameer from Bellarmine accidentally accessed the Sectionals tournament before TI, and thus chose to sit out.
This is true. Short version:
1) High school players are allowed to write for Division I SCT.
2) That doesn't work for Texas Invitational, but we didn't catch that until after we hired Sameer.
3) NAQT would like to commend Sameer, his coach, and his teammates for their integrity in promptly reporting the situation to us.
We will continue to allow high schoolers to write questions for NAQT, but we will discuss revisions to our policies over the summer. One option we will consider is no longer allowing active high schoolers to write for Division I college play (so that we no longer have to track field updates for events like the Texas Invitational).
Horned Screamer wrote: I would like an explanation of whether or not there is truth to the rumors that Sameer from Bellarmine accidentally accessed the Sectionals tournament before TI, and thus chose to sit out.
Sameer Rai did accidentally access a page for the Division I SCT set prior to the Texas Invitational. He promptly informed NAQT of this fact and volunteered to sit out the tournament. We consider him to have acted professionally and appropriately. NAQT has since created additional writer groups for high school player-writers who intend to compete on collegiate sets.
In general, NAQT has not had a disproportionate number of problems with its high school writers, and we have not previously considered a general ban on precollegiate writers as a response to these security issues. I will, however, add this to the list of things for the members to discuss.
Post by AKKOLADE » Wed Mar 20, 2013 3:41 pm
Horned Screamer wrote: I frankly think that, at this point, NAQT should stop hiring high schoolers.
I encourage NAQT to take this stance.
Aren't there high schoolers who play events on literally every level of quizbowl but D1 ICT? What's the benefit to letting them in to write for one tournament when the payoff is that it makes your organization look way more unprofessional and gets people asking questions about when the next time one of these kids will be an Amit Bilgi of the future and destroy an HSNCT?
The Bold Ideas of Bernie Sanders (I-VT)
Post by The Bold Ideas of Bernie Sanders (I-VT) » Wed Mar 20, 2013 3:56 pm
Horned Screamer wrote: Aren't there high schoolers who play events on literally every level of quizbowl but D1 ICT?
That has actually happened before.
Adam Sperber
Hickman '10, Northwestern B '14
" 'Yay, more Adam Sperber' --Nobody " --Cody Voight
None of NAQT's high school writers currently compete on our middle school sets; a very small number (two?) compete on Division I questions.
What's the benefit to letting them in to write for one tournament when the payoff is that it makes your organization look way more unprofessional and gets people asking questions about when the next time one of these kids will be an Amit Bilgi of the future and destroy an HSNCT?
The primary benefit to NAQT comes in letting high schoolers write for middle school tournaments: That gives them practice writing questions, which will stand them in good stead as they get older and can write for other events. It also frees up our more senior writers to produce harder questions for other sets. The benefit to NAQT specifically in allowing high schoolers to write for Division I collegiate sets is quite small, since the number of high school writers who can produce usable material at that level is very limited (but not zero).
I suppose I'll add that I am a little bit surprised that this was the first issue brought up in this thread; I hadn't previously thought of high school writers as contributing more than their share of security issues (or other kinds of issues) to quiz bowl; is there more evidence from non-NAQT organizations that NAQT should consider?
I mean, Amit ruined PACE in 2010 due to cheating, there have been many other instances of high schoolers cheating or attempting to cheat at other events (remember Neil Sammader, there have been local tournaments I've been at where people cheated, I'm sure lots of people have lots of stories). The bottom line is that there are a lot of high schoolers who will cheat given the opportunity, even from established programs like Wilmington Charter a single year after they won their titles. Separately, I think there's been too much willingness to not set boundaries between high schoolers and adults, and I think hiring high schoolers to write for a company that is supposed to be adults presenting a product to other adults is really counterproductive to that. There have always been people wondering if all these high schoolers playing NAQT are cheating, and these incidents demonstrating just how possible it is for high schoolers to cheat is now suddenly making this a much bigger issue, so why would you want to risk it?
Separately, even though NAQT's current writing staff doesn't play all of those MS sets, it's still the case that middle school questions do get run at some high school events, no? Is it not a very reasonable assumption to say in the future a high school writer could play on one of those sets unless NAQT stops selling them to HS events?
Last edited by Sen. Estes Kefauver (D-TN) on Wed Mar 20, 2013 4:24 pm, edited 1 time in total.
2008-09 Male Athlete of the Year
Post by Cody » Wed Mar 20, 2013 4:20 pm
These accusations have been around for nearly 3 years now, and at one point, we were told that some kind of logs were reviewed and Andy Watkins had not cheated. Was the earlier response untrue? Why did it take so long to truly investigate whether or not Andy Watkins had cheated (especially when we all knew it to be true)?
Cody Voight (he/him), VCU '14. I wrote lots of science and am an electrical engineer.
VCU Tournament Director '13-'17. HSAPQ President '15-'16. ACF Treasurer '19-'20. ACF Nats '21 TD.
Hero of Socialist Quizbowl Labor (NSC '14). "esteemed colleague" of Snap Wexley, ca. 2016. Stats Hero (Nats '16). "stats god in the flesh, the ominous Indominus Rex" (Winter '20).
Quizbowl at VCU
Post by Stained Diviner » Wed Mar 20, 2013 4:49 pm
I don't think the issue is allowing high school students to write. As we know from this thread, college students are perfectly capable of cheating, and as some of us know from various experiences, coaches are perfectly capable of cheating.
If people think it's really an issue, NAQT probably can set the high schoolers up so they have very limited access to Ginseng and can only see their own questions. This would make it difficult for those high school students to finish off a set because they wouldn't be able to see if something has already come up, but that seems like it would be a fairly minor loss to the writers and NAQT.
I still think it is OK for NAQT to pass off the questions as being written by adults because they are being edited and compiled by adults, and the overwhelming majority of them are written by adults.
The big problem here is that NAQT made it too easy for too many people to cheat, and some of those people cheated. NAQT claims it has addressed this issue and will continue to address this issue. Of course, at this point people should be asking questions about that and making suggestions.
Leucippe and Clitophon wrote: I still think it is OK for NAQT to pass off the questions as being written by adults because they are being edited and compiled by adults, and the overwhelming majority of them are written by adults.
So why risk the liability if most are written by "adults" ?
At this point, they certainly have to consider the liability seriously. If writing for NAQT as a high school student does not give you access to questions you might play on, then it's possible that the liability is low.
I'll add that anybody who writes for NAQT should not be playing on a middle school set. The reason you use a middle school set is because you have a field that would do poorly on an A Set. One of the reasons NAQT pays you money to write questions is that you have enough of a sense of the canon to be able to destroy an A Set. That should be simple policy.
vinteuil
Post by vinteuil » Wed Mar 20, 2013 5:59 pm
Leucippe and Clitophon wrote: If people think it's really an issue, NAQT probably can set the high schoolers up so they have very limited access to Ginseng and can only see their own questions. This would make it difficult for those high school students to finish off a set because they wouldn't be able to see if something has already come up, but that seems like it would be a fairly minor loss to the writers and NAQT.
I definitely support this (as a high school writer for NAQT), partly because we tend to get emails with big lists of what kind of questions are needed to finish off a given set anyways.
Last edited by vinteuil on Wed Mar 20, 2013 6:35 pm, edited 1 time in total.
Post by marnold » Wed Mar 20, 2013 6:30 pm
Hey, it's obviously shitty this announcement had to be made, but I'm really glad that NAQT is taking all these things seriously and making things public. Even though it's a little embarrassing in the short term, it's showing (genuine) good faith that makes me trust NAQT as an organization.
Post by Adventure Temple Trail » Wed Mar 20, 2013 6:58 pm
It is possible for a high schooler to play Division I ICT if they are dual-enrolled at a college or university (as Tommy Casalaspi was when he played Division I in 2011 for VCU); therefore, it is indeed possible for a non-hypothetical high school NAQT writer to play any difficulty level from 3 to 9, and all high school writers should be barred from any access to all questions from levels 3 through 9. It invites too many opportunities for mistake or malfeasance to let anyone who is in high school access any information about any questions which can be played by any person who is still in high school. I'm personally fine with high schoolers writing for middle school sets, if it meets the desired goals of making middle school sets happen and preventing the idiotic, patronizing practice of running high school tournaments on middle school questions (a practice which, for similar reasons as above, has to be outright banned).
I am also of the opinion that DI-eligible writers should not be able to see the Topics page for the Division II SCT if they write for Division II. (Alternately, the Topics page for multi-division sets should be altered with computer wizardry such that the topics from DI-converted questions are not visible at all.) This may make the completion of the set somewhat less efficient, but it prevents people from reverse-engineering the set of answers in the set of the other division and running off with them for nefarious purposes.
It seems as though the elimination of the ability to look at the first-40-character sample solves most of the other issues with the way these people went about things. But I strongly suggest that NAQT make a serious effort to redouble its security efforts, even if it means running through each week's logs every week (I'm not sure how labor-intensive a process this is, actually).
Steeve Ho You Fat
Post by Steeve Ho You Fat » Wed Mar 20, 2013 9:39 pm
RyuAqua wrote: I'm personally fine with high schoolers writing for middle school sets, if it meets the desired goals of making middle school sets happen and preventing the idiotic, patronizing practice of running high school tournaments on middle school questions (a practice which, for similar reasons as above, has to be outright banned).
As someone who wrote for NAQT while I was in high school, I agree fully with this. I didn't even realize that people were running high school tournaments on middle school questions.
See, I was confused about this: previously I was told that anyone playing one division of SCT couldn't write for either both to prevent nefarious things and to avoid problems in case of, say, a combined field. This year, however, both Shan and I had questions in the DII set and could see all of it (I had to bite my tongue a couple of times to keep from mentioning 45 questions I wrote while around my DII teammates).
Joe Nutter
PACE Treasurer
Walnut Hills High School '11
Urech hydantoin synthesis
Post by Urech hydantoin synthesis » Wed Mar 20, 2013 9:46 pm
RyuAqua wrote: and all high school writers should be barred from any access to all questions from levels 3 through 9. It invites too many opportunities for mistake or malfeasance to let anyone who is in high school access any information about any questions which can be played by any person who is still in high school
Would this include the television and state-specific sets that NAQT produces? It seems that high schoolers contribute a non-trivial amount of questions to them.
Ben Zhang
Zucker School of Medicine at Hofstra/Northwell '23
Columbia University '18
Ladue Horton Watkins HS '14
Christ, I Know wrote:
Yes, due to the interconversion between TV questions/state-specific questions and other sets such as A-sets.
Plan Rubber wrote: I didn't even realize that people were running high school tournaments on middle school questions.
There are a handful of these events; the usual case is the previous year's MSNCT reused for a 9th-grade or JV division in an area where all of NAQT's introductory high school questions have already sold out.
STPickrell
Location: Vienna, VA
Contact STPickrell
Post by STPickrell » Wed Mar 20, 2013 11:04 pm
RyuAqua wrote: It seems as though the elimination of the ability to look at the first-40-character sample solves most of the other issues with the way these people went about things. But I strongly suggest that NAQT make a serious effort to redouble its security efforts, even if it means running through each week's logs every week (I'm not sure how labor-intensive a process this is, actually).
If you have a way of associating a given user with an IP address, then it'd be reasonably easy to script (assuming you have shell access and perl/Ruby/PHP/language of choice.)
It might get trickier if two roommates are both NAQT writers, though.
Shawn Pickrell, HSAPQ CFO
Post by jonah » Wed Mar 20, 2013 11:17 pm
RyuAqua wrote: But I strongly suggest that NAQT make a serious effort to redouble its security efforts
We are doing so.
RyuAqua wrote: even if it means running through each week's logs every week (I'm not sure how labor-intensive a process this is, actually).
We are doing this.
STPickrell wrote: If you have a way of associating a given user with an IP address, then it'd be reasonably easy to script (assuming you have shell access and perl/Ruby/PHP/language of choice.)
Access to our internal site requires authentication, and the authenticated username is included in our logs.
Jonah Greenthal
National Academic Quiz Tournaments
Post by jonah » Thu Mar 21, 2013 12:06 am
RyuAqua wrote: I am also of the opinion that DI-eligible writers should not be able to see the Topics page for the Division II SCT if they write for Division II. (Alternately, the Topics page for multi-division sets should be altered with computer wizardry such that the topics from DI-converted questions are not visible at all.) This may make the completion of the set somewhat less efficient, but it prevents people from reverse-engineering the set of answers in the set of the other division and running off with them for nefarious purposes.
The Topics pages now pretend that any topics referenced in questions used in packet sets the user can't see don't exist. (Sorry if that's hard to follow. Basically: Go through every topic that meets the search criteria. Look at the question referencing that topic. Look at every packet set in which that question and all the questions in its family [that is, the question from which it is derived, if any, and that question's parents, etc.; and all questions that derive from it, and their children, etc.] is used. If any of those packet sets is supposed to be hidden from the user, then the display of the topic is suppressed.)
quizbowllee
Post by quizbowllee » Thu Mar 21, 2013 1:22 pm
So, all of these issues have been internal. My question is, is NAQT vulnerable to an outside hack? The clientele in general is extremely smart and tech-savvy. And, unfortunately, it seems that not everyone involved in quiz bowl is completely trustworthy.
With the recent hacks of major corporations and businesses and the subsequent compromise of credit card numbers, etc., I can't help but wonder if a kid could hack NAQT and compromise the questions to an event? Are there measures in place to keep this from happening and/or detecting a breach when and if it occurred?
Lee Henry
AP English Teacher
Quiz Bowl Coach - West Point High School (Cullman, AL)
President - Alabama Scholastic Competition Association (ASCA)
Post by Sen. Estes Kefauver (D-TN) » Thu Mar 21, 2013 1:44 pm
Did anybody from NAQT know that there were a bunch of vague allegations going around in back channels (IRC and the like) that Andy had cheated after the 2010 ICT, or did anybody ever contact NAQT naming names about Andy? If so, can NAQT give an honest answer about whether or not they thought the reports were serious, or if part of the reason they took until now was because NAQT decided it wasn't very likely one of their up and coming editors would do this? Did Watkins only get caught now because of more intense security sweeps accidentally cropping him up, or was NAQT specifically looking at his activity?
Because I've heard nothing but awful things from day 1 about Ginseng being built with massive security holes that, this is showing, relied on believing that NAQT's writers would do the right thing if they were, say, given the option to see the first 40 characters of every question. I'm piggybacking on Cody's post in the other thread here, but can R. explain why it is that for so many years NAQT was so lax about this whole thing?
Post by rhentzel » Thu Mar 21, 2013 2:53 pm
SirT wrote: These accusations have been around for nearly 3 years now, and at one point, we were told that some kind of logs were reviewed and Andy Watkins had not cheated. Was the earlier response untrue?
The short answer is that there was an earlier review--which concluded that Andy had not had improper access--but that review fell short of professional standards and didn't find things that could have been found.
Why did it take so long to truly investigate whether or not Andy Watkins had cheated (especially when we all knew it to be true)?
As best as I can tell, both from my memory and my e-mail archive, there was no suggestion after the 2009 ICT that anything untoward had occurred. In late 2010 NAQT became aware of rumors/suggestions regarding the 2010 ICT (as best as I can tell there was no formal accusation ever made with NAQT); as a result, we checked the server logs for the pages that were known security holes (primarily "category.jsp"), but didn't find any inappropriate access. We were also aware of suggestions that Andy may have used a different writer's account; that was more thoroughly investigated (as it seemed more plausible), but also turned up nothing (that negative finding was corroborated by the more recent review). We also checked a number of types of "real hacking" (firewall logs, database access, etc.) and didn't see any evidence of intrusion.
The role of the "question-by-writer" and "questions-in-processed-file" pages as security holes became clear later, mid 2011. NAQT did continue to hear rumors of inappropriate access throughout 2011 but, as best as I can tell from my e-mails, they all specifically pertained to the 2010 ICT, the investigation of which had been deemed closed.
So why did they come to light in 2013? Well, fundamentally, I think finding and proving the first case opened our eyes; if one person really was doing this, others might be as well, so we started a new review which was done much more thoroughly and more generally. Other contributing factors include NAQT's gradual evolution into a company that has more time (and money) to pay for things that are not directly relating to writing and editing questions and acquiring a new member like Jonah Greenthal who had an interest in improving NAQT's technical infrastructure (and the skills to do so).
NAQT's earlier review was simply not done as it should have been; we looked at specific pages known to be vulnerable (and a lot of things that turned out to be dead ends), but we had the data and the technical skills to have found this in May 2009 (let alone 2010), but we didn't devote enough time to it. In essence, we did enough to be able to tell people who suggested the existence of a problem, "We checked that method you thought might have been used, but we can assure you it wasn't."
As Chief Technology Officer, the incomplete nature of the first investigation was my fault, and I am embarrassed to have been wrong and sorry to have had NAQT's other members rely on my findings. And--of course--most sorry to have had the integrity of three consecutive collegiate championships compromised. NAQT will be spending more time and money in the future on investigating any accusations brought to its attention, and we have already committed to spending more time and money on pro-actively closing holes and security monitoring so that such accusations may never need to be made.
Post by jonah » Thu Mar 21, 2013 2:56 pm
quizbowllee wrote: So, all of these issues have been internal. My question is, is NAQT vulnerable to an outside hack? The clientele in general is extremely smart and tech-savvy. And, unfortunately, it seems that not everyone involved in quiz bowl is completely trustworthy.
It would be foolish to say "no, we're completely invulnerable", but my belief is we are not at serious risk of compromise. NAQT doesn't have anything that would be of interest to hardcore criminals, so I assume the question is primarily whether NAQT is vulnerable to attacks by quizbowl people who do not work for it. Certainly all precedented attacks on NAQT have relied on exploiting existing access beyond what ought to have been possible. That is not to say that other, external attacks are impossible, but I have conducted—and continue to conduct—penetration testing, none of which has found vulnerabilities thus far. We have made significant upgrades to our internal and external security, and will continue to do so in the next few months, over the summer, and ongoing. We have also taken precautions against social engineering attacks, and would appreciate further feedback (in private to [email protected]) on such issues.
(edited to remove unintended qualification)
Last edited by jonah on Thu Mar 21, 2013 3:05 pm, edited 1 time in total.
merv1618 wrote:
The most direct answer to this is that NAQT perceives the primary security risks in dealing with high schoolers to lie with players, rather than writers. Charlie did produce a lengthy list of malfeasance by high schoolers, but I believe none of those incidents involved writers. If anything, I would say that the phenomenon of which NAQT needs to be wary is players (at any level, but in this context, high schoolers) attempting to access packet sets by setting up fake tournaments or impersonating coaches or moderators. And we do consider that to a problem that needs to be addressed.
Post by conker » Thu Mar 21, 2013 3:06 pm
Out of curiosity, what were the warning signs about Andy that made people suspicious in the first place? As his teammate on one of the ICT teams, I hadn't the faintest idea that he was cheating, but of course by that point I was already mostly out of the quiz bowl loop.
Dennis Sun
Shanghai American School '06
Stanford '15
Sima Guang Hater
Post by Sima Guang Hater » Thu Mar 21, 2013 3:13 pm
conker wrote: Out of curiosity, what were the warning signs about Andy that made people suspicious in the first place? As his teammate on one of the ICT teams, I hadn't the faintest idea that he was cheating, but of course by that point I was already mostly out of the quiz bowl loop.
His statline at ICT was far better than at ACF nationals. That usually doesn't happen unless you're a geo or trash specialist, neither of which he was.
Eric Mukherjee, MD PhD
Brown University, 2009
Perelman School of Medicine at the University of Pennsylvania, 2018
Medicine Intern, Yale-Waterbury, 2018-9
Dermatology Resident, Vanderbilt University Medical Center, 2019-
Writer, NAQT, NHBB, IQBT, ACF, PACE
bradleykirksey
Post by bradleykirksey » Thu Mar 21, 2013 3:17 pm
I've never seen Andy play, wasn't in the QB loop when he was at ICT, and I've never even met the guy. My best guess would be the 17 powers, 22 tossups, and 0 neg answer line at the 2011 ICT. You weren't there for that, but he was the 14th highest scoring players overall, the only one of the top 46 without a neg, the only one of the top 15 with fewer than 3 negs, and had an insane number of powers (17) to PPG (55.56). Matt Bolinger was the only person with more powers (18) and he had 5 negs and not twice as many PPG, but close (94.64). The 17 powers to 22 tossups is probably the most worrisome I would guess. It would be particularly interesting to see someone with a ton of powers, not that many tossups, and no negs.
Of course, like I said, I wasn't there in 2011, I've never met Andy, and I don't know the details of where he got those answers. For all I know, he told people and it got out. I'm just speculating.
Bradley Kirksey
Mayor of quiz bowl at the University of Central Florida (2010-2015)
The club at Reformed Theology Seminary Orlando (2017 - 2055??)
RyuAqua wrote: It is possible for a high schooler to play Division I ICT if they are dual-enrolled at a college or university (as Tommy Casalaspi was when he played Division I in 2011 for VCU); therefore, it is indeed possible for a non-hypothetical high school NAQT writer to play any difficulty level from 3 to 9, and all high school writers should be barred from any access to all questions from levels 3 through 9. It invites too many opportunities for mistake or malfeasance to let anyone who is in high school access any information about any questions which can be played by any person who is still in high school. I'm personally fine with high schoolers writing for middle school sets, if it meets the desired goals of making middle school sets happen and preventing the idiotic, patronizing practice of running high school tournaments on middle school questions (a practice which, for similar reasons as above, has to be outright banned).
A player in Tommy Casalaspi's position would be put in a special writer group that prohibited access to questions in the DI set. In addition, NAQT currently tracks every single question "exposure" (through writing, editing, online browsing, and tooltip display) so that we can say "Person X saw question Y on date Z." That allows us to double-check that players registered for tournaments (or special cases like a hypothetical 2013 Tommy Casalaspi) have not seen the questions in a set even if they looked at the questions before we set their writer group appropriately.
NAQT agrees with the principle that players should not--as a technical matter--be to able access substantive information about questions on which they end up playing; most of the time that involves simply disallowing certain page hits, but it may also involve tracking questions. For instance, high school player X might see middle school question Y in middle school packet Z, but then an editor decides it's too hard and kicks it up to high school. What was previously valuable access facilitating set completion, has become a security hole. But, since we track every question's exposure, we can catch that and refrain from using that question until player X has moved beyond high school sets.
We are now (and have been for a while) reviewing logs weekly and will also do special reviews immediately before national championship tournaments.
To be clear, NAQT has separate writer groups to prevent a case like, "Washington high school writer A writes a question for Oregon set B, but then that same question also shows up in Invitational Series #200A, which is played in Washington."
A question by, for instance, a collegiate writer could be shared in such a manner, but our system would not allow writer A's question to be built into Invitational Series #200A (because writer A can write for set B, but not for IS #200A).
The Quest for the Historical Mukherjesus wrote:
Also he wasn't putting up stats that were THAT much better than his teammates (usually worse) at all the regular tournaments that year. It was like that ICT was his sudden breakout performance.
Post by The King's Flight to the Scots » Thu Mar 21, 2013 3:36 pm
Eric and Dees have given a good explanation, but to elaborate: he had never played at anywhere near that level until the 2010 ICT, then he had a mediocre performance at 2010 ACF, then he played amazingly well at the 2011 ICT again, and then he again played poorly at 2011 ACF. Basically, those two statlines were incredibly anomalous compared to everything he had played previously, and it couldn't have just been improvement because he didn't play that well afterwards either. He was also an NAQT editor and would have had access to some of the well-known security holes.
Post by Stained Diviner » Thu Mar 21, 2013 3:43 pm
The statements in this thread contradict the NAQT announcement and Andy's statement. The announcement claimed that there was no statistical evidence of cheating, yet people who paid attention to the statistics claim that things looked very suspicious.
My guess is that this is because NAQT used a simplistic definition of statistical evidence.
Horned Screamer wrote: Did anybody from NAQT know that there were a bunch of vague allegations going around in back channels (IRC and the like) that Andy had cheated after the 2010 ICT
Yes, we knew. In fact, I would go so far as to say that we knew there were more than "vague allegations," and something like, "Some people think Andy may have had prior access to question content in this fashion (hitting these pages, using another account, "hacking"). I won't pretend that every NAQT member is on these forums and in the IRC chatroom every day, but we do try to stay connected with the game's online community.
. . . did anybody ever contact NAQT naming names about Andy?
I don't know exactly what level of "formal contact," you have in mind, but I have two e-mails from people in 2010 naming Andy.
If so, can NAQT give an honest answer about whether or not they thought the reports were serious, or if part of the reason they took until now was because NAQT decided it wasn't very likely one of their up and coming editors would do this?
At the time, I thought that we were treating them with appropriate seriousness; we went through our server logs (and firewall logs, etc.) looking at the specific avenues of access that people had suggested (and others that we knew about at the time). I, personally, didn't see damning evidence of prior access in the tournament performances themselves; Andy played well, but his year-to-year improvement (both on NAQT questions and at other tournaments) seemed well within the normal variation of dedicated quiz bowlers. Certainly, at no point do I remember anybody saying, "This is just too ridiculous to investigate" or "NAQT shouldn't insult its editors by checking on them" or anything like that.
Did Watkins only get caught now because of more intense security sweeps accidentally cropping him up, or was NAQT specifically looking at his activity?
A little of both, actually. After the first incident came to light, we specifically reauthorized a review of this particular case, but the approach we used that discovered the other two cases would also have turned it up.
The short, and probably unsatisfying, answer really is that developing an online production platform that has strong security and tracking while remaining flexible enough to allow many different classes of writers the access they need to effectively work on many different classes of packet sets is time-consuming and expensive. At some point, however, NAQT needs to be able to trust its writers. Even if we lock our internal systems down perfectly, a non-playing writer could screenshot a bunch of questions and sell them to the highest bidder. NAQT didn't really see it as a question of, "We don't trust our writers, so we need to prevent every possible form of attack," but rather, "We have to trust our writers, so we might as well give them efficient tools to do their job." But, as events have proven, we drew the line in the wrong place and are now needing to move it (and rethink other key aspects of question security).
Tees-Exe Line
Post by Tees-Exe Line » Thu Mar 21, 2013 4:16 pm
rhentzel wrote: After the first incident came to light, we specifically reauthorized a review of this particular case, but the approach we used that discovered the other two cases would also have turned it up.
By "the first incident," are you referring to Shantanu Jha, Josh Alman, or someone else? Are those two cases and the three revealed yesterday all the due to the same security hole?
Marshall I. Steinbaum
Oxford University (2002-2005)
University of Chicago (2008-2014)
University of Utah (2019- )
Get in the elevator.
Post by Important Bird Area » Thu Mar 21, 2013 4:20 pm
R. was referring to this incident, not this incident. (The latter case had nothing to do with the security of NAQT's website; rather it involved an NAQT editor inappropriately revealing question content to players who were about to play the 2012 ICT.)
Tees-Exe Line wrote:
That was unclear; sorry. By "first incident," I meant this one. Broadly speaking, I would say that it and the ones announced this week were essentially the same class of security holes.
The (even) earlier incident involving an NAQT editor and the 2012 ICT was qualitatively different in that the editor enjoyed legitimate access to the questions involved. But he may have shared that material with other players under the guise of helping them prepare for the tournament.
bt_green_warbler wrote: R. was referring to this incident, not this incident. (The latter case had nothing to do with the security of NAQT's website; rather it involved an NAQT editor inappropriately revealing question content to players who were about to play the 2012 ICT.)
Well, an "editor" who was not editing the 2012 ICT! And thus whose access to it should have been curtailed.
Even if the nature of the security breach wasn't the same in each case, it seems hard to credit the late 2013 revelation R. mentioned above. If NAQT considers Shantanu's cheating an entirely separate phenomenon that did not herald this year's revelations, how was it that Josh's actions first became known to you?
Post by Margo » Thu Mar 21, 2013 4:44 pm
The security breach here was the fact that NAQT trusted a compulsive liar, not that he had access to the tournament. These cases are obviously unrelated.
Formerly from Chicago
Currently at University of Michigan
Tees-Exe Line wrote: If NAQT considers Shantanu's cheating an entirely separate phenomenon that did not herald this year's revelations, how was it that [other actions] first became known to you?
We received a tip from a source who wishes to remain anonymous.
bt_green_warbler wrote:
My connecting them is frankly related to the experience of handling Shantanu's case last year, based on which I came away quite ready to believe the longstanding rumors about Andy. That is also why I asked in the other thread for some indication that ALL the NAQT editors give some indication of understanding that the cheating problem is endemic and not a matter of isolated actors, compulsive liars though some of them are.
Post by grapesmoker » Thu Mar 21, 2013 5:59 pm
Not that I want to turn this into a technical discussion, but it seems like these are all vulnerabilities that were generated by a bad security model. It seems to me that if you have a properly role-based model, you shouldn't be having these problems in the first place. Obviously I have no idea what Ginseng looks like at the code level, but my impression from all this talk and previous conversations is that it's an old codebase that's sort of accreted throughout the years. It might be a good idea to think seriously about a wholesale redesign with a proper role-based security model.
SmallerMegalomaniacalPandaOnAbsinthe
Location: Falmer, East Sussex
Post by SmallerMegalomaniacalPandaOnAbsinthe » Thu Mar 21, 2013 6:56 pm
Margo wrote:
A compulsive liar? Perhaps I am out of line here, but I would ask that you maintain some basic modicum of human decency in your posts. I do not think it too much to ask that we avoid libelous rubbish on these forums. You have made a misrepresentation (whether innocent or motivated by some animosity) regarding a person you evidently did not know very well.
User was warned for telling other people how to post. --Mgmt.
Ketan Jha
Whitman '12
Sussex Law School '15
Post by AKKOLADE » Thu Mar 21, 2013 7:14 pm
Maybe we should actually discuss cheating cheaters that cheated in the thread about this issue.
Auroni
Location: Urbana
Post by Auroni » Thu Mar 21, 2013 7:30 pm
I'm not part of the University of Chicago quizbowl team, which had to deal with Shantanu for many years, but even I know that he's a compulsive liar. From having him tell me that he's "vacationing in India" to avoid question writing responsibilities for sets that I was part of, to completely ridiculous and transparent bullshit like this: viewtopic.php?p=172611#p172611, to him presumably covering up for his actions in the cheating scandal, it's not that hard to see.
Last edited by Auroni on Thu Mar 21, 2013 8:00 pm, edited 1 time in total.
Auroni Gupta (she/they)
UIUC
SmallerMegalomaniacalPandaOnAbsinthe wrote:
I was on the University of Chicago quizbowl team for almost the entire time your brother was, and admirable though it may be to assert family loyalty and love when he's publicly attacked it won't get you very far with me or in quizbowl-land in general. Shantanu is a serial, compulsive liar, as well as a cheater. When confronted with incontrovertible evidence that he'd accessed the 2012 ICT questions in advance, many times, despite having nothing to do with the tournament, he denied it repeatedly, serially, in such a way that he thought he was defending himself based on one set of evidence only to have that iteration eviscerated by the next. Furthermore, he has never admitted his behavior or apologized to the team whose performance he rendered suspect.
That cheating scandal was the last straw after YEARS of lies. He lied to the packet-writing team for ACF Regionals 2012, saying that he'd written and submitted a packet when he had not in fact. He lied to me as editor of Peaceful Resolution 2012, claiming to have 1. written many questions that he could never manage to produce, and 2. telling other writers that he would cover their share so not to worry about writing their full commitment. He claimed to be too busy to discuss editing responsibilities for PR because he was preoccupied with GJR-GARCH models. He claimed to have translated a volume of Eugenio Montale's poetry, and to have been so moved by a Simone Martini altarpiece that he stared at it for four hours in tears.
In short, your brother is a compulsive liar, one who had a disastrous effect on the success of my quizbowl team across many years. I was enraged by his conduct at ICT 2012 and by the seeming tolerance with which it was greeted in some quarters, which is what motivates the concern I expressed above about the pattern in NAQT's behavior. The most likely outcome of what I've written here is most likely simply to engender hatred in yet another member of your family, but that's worth it for at least a start on the comprehensive catalog of his dishonesty and general misbehavior.
I have never understood your brother--his obvious intelligence wasted on a quixotic effort to alienate everyone, his inability to succeed academically despite an amazing knowledge of esoteric subjects, his need to assert a superior cultural and intellectual status when he has the raw material to impress the world without resort to lies. I hope in time he will be able to achieve his potential.
I think this has already been covered pretty well but I was on or associated with the Chicago team the entire time he was here and I can count the number of lies he has told on more than one hand. Obviously I don't know the reason behind or level of compulsion associated with these lies but I don't think it's an unfair accusation.
Return to "College area archives"