Four Features 

Kit Nicholls
with Carl Sable

February 2025
Issue 1

This interview has been edited for clarity and concision…

Kit:
I'm sure you know that there's a lot of anxiety about how ChatGPT is going to affect the ability of faculty to evaluate written work by students. Given your understanding of the technology itself, to what extent should faculty worry or not worry?


Carl:
This is something I've given a lot of thought. I think part of the answer depends on how much further the technology will advance, which I don't think anyone knows. Experts predict wildly different things. I've heard predictions that within a couple of years, it's going to be able to do everything—pass the Turing test, whatever. I've heard other experts predict that it's close to a ceiling, that it's gotten very advanced, but it's not going to get much more advanced.

Even in its current state, I think it can really help students with at least first drafts of essays, and whether that would be considered cheating depends on different faculty perspectives. But it's already, in its given state, something that we have to think about because the student can get a first draft of an essay, and then edit it, or not. In certain classes that is potentially very problematic.

Now, this question also depends on what we consider to be “written work.” I mean, from my own perspective, I also have worries about using generative AI systems to produce code. In my classes, students have to write programs. In the classes that I teach, specifically, the systems are definitely not at a point where the code that is generated is enough, but perhaps in some intro programming classes, it could be.

I have not tried to restrict the use of chat GPT–when it comes to writing code I don't think it's something that I could enforce. I just ask students to indicate if and how they've used it when they do an assignment. And I think the initial impression last semester is that the students–and there might be students who used it without indicating it–but the students that indicated they used it did not seem to be helped much. I've noticed sometimes it leads to bugs and to errors.

But, for essays, what are college admissions going to do? Admissions are moving away from standardized tests and more towards essays, and now essays can be written for you. People who get more help on essays were always favored, but now it's just a different type of help. And I do think that in humanities classes where essays are important, it's already at a point where essays have to be written in-person to know that students are writing them themselves. It's a big change because this technology wasn't like this a year and a half ago.

Kit:
There are definitely dangers in the humanities classroom because, much like the lawyer who had AI writing his briefs for him, the AI makes stuff up. I got a student paper last semester where chatGPT invented a character in a novel, and the way it invented the character played to the most boring way of imagining what the book was. It's like, oh, this is a Mexican character so it connects it to some values of diversity and it's just the blandest most vanilla thing. So, next question: what's different between the way a Large Language Model like chat GPT “thinks” and the way we humans think? And what has the development of generative AI taught us about where ideas come from?


Carl:
Some experts will disagree with me, but I don't think generative AI teaches us anything about where ideas come from. People ask questions like, do these machines think? To what extent are these machines conscious, self aware etc.? My best answer is, the degree to which an LLM is actually thinking or conscious or aware is exactly zero, like a rock. Algorithmically, LLMs predict one token at a time based on everything that has been in the dialogue so far. They predict the best or most likely next token based on a probability distribution, and they're not deterministic. They're pseudo-random, so you won't get the same thing every time. 

That is not how a human thinks. Some would claim our free will itself is an illusion and we're just next token predictors, but I don't believe that. I believe consciousness is a thing–I don't know what it is and don't think anybody knows what it is–but I think we have thoughts, and we don't know what thoughts are. But the thought is a complete thought, then we decide how to express the thought in language. But that's not what an LLM does. An LLM produces one token at a time until it predicts an end of sequence token. People might claim that the encoding the input has, and the weights in the model, are somehow representing a real worldview that is a thought, and that even though algorithmically it's predicting one token at a time, it is doing it in such a way to express a thought. I think that’s nonsense. It's a Turing machine. And that is not consciousness. I find what algorithms can produce extremely impressive, but it’s a very different thing.


Kit:
We’re at a point where some of the big generative AIs are able to produce video, audio, etc., but the variety of sense perceptions that funnel together to produce the thought–it’s a far more complex system than the current AIs seem to capture. If you talk to people in the School of Art or Architecture and ask them what it's like to think, it is often prelinguistic or alinguistic. There's a lot of thinking that doesn't happen in words. And I don't know that we have any way to account for that in our present understanding of how LLMs work.


Carl:
I was surprised—like I said I might be in the minority—but I'm not in as small a minority as I thought. There was a survey of NLP experts who have published in the last couple years, and there are many people who agree with this point of view—that LLMs don't think, are not conscious, and can't be. Some of them believe the missing component is things like audio and video, but that wouldn’t really add that much from my point of view. I don't think that if you give an LLM video and audio input, that it's going to change anything–it's still digital, and all it's getting is bits. So I don't really care if the bits represent what we consider language or video or audio. It doesn't have qualia, in my opinion.

Qualia was a concept that confused me for years and I used to think it was just a made up word describing something we don't actually have. I disagree with that now that I understand better what it is. We have sensory perception and it produces some sort of feeling that some philosophers call qualia which, whatever it is, we don't know what it is and, look, I'm an atheist. I'm a scientist. I don't believe there's any magic involved, but just because there's not doesn't mean it is a digital algorithm. I think whatever our brain is doing is something else that we just don't understand.


Kit:
I also feel like the existence of synesthesia, alone, suggests already just how much we don't know about the way that our senses interact. Like, I've got a friend who's a composer, and anyone he knows well, he associates a key and a color with that person. And that's just part of how his mind is.


Carl:
Yeah, I've heard of that sort of thing. And, you know, I think almost everyone would agree that a person who is blind and deaf can still be conscious, right? So it's not just the sense, but it's even without it, people have these conscious feelings and ideas. Even if the sense isn't there. There's something they're experiencing in their mind. There are people who don't believe in this and, again, I used to think the brain is basically a giant neural net, and not even really a single thing. I used to say the mind was a kind of illusion. I don't think it's an illusion anymore.

I think there are things in physics that we don't know yet. And I do think, again, that there is consciousness that we have without knowing what it is. I think it is a real physical thing not in the sense that it has mass but in the sense that there are laws in our universe that allow for it, and we have it and we don't know what it is.

I think ChatGPT is not conscious but is very impressive. Eventually, colleges are going to have to come up with policies to account for it, and I don't know what they should be. But it only appears to do a thing that looks like being a student.


Kit:
Yes, because it does this thing that resembles being a student, we're in the throes of a weird representational problem: What is being a student versus what looks like being a student. So that leads us to the next question. If you had all the time in the world—by which I mean all the time in your students’ schedules—how would you teach AI ethics?


Carl:
So I'll take a quick excursion to say what I do in my NLP [Natural Language Processing] class. I have a set of lessons toward the end of the course on the ethics of AI and NLP. I tell the students that I have no formal background in philosophy and that ethics is at least considered by some to be a subcategory of philosophy. But I'm interested in it as you can probably see, and so I include it in my courses.

But maybe even without all the time in the world, if there was an appropriate person in HSS that had a background and interest in AI ethics I would love to develop a course—to co-develop and co-teach. I’ve looked at other academic institutions and there's basically nothing on it. We have a course in the ethics of computer science but it's taught by a non-technical person. Sometimes you see a course in other schools taught only by a technical person. There's almost nowhere where you have people with backgrounds in ethics and people with backgrounds in AI co-developing and co-teaching, and I would love to do that. I think it would be great to co-develop and co-teach a course that's modified over time to keep up.

Kit:
Great. So, I have a guess about how you're going to answer this next question: has generative AI been helpful to you and your own research processes? Do you know of any colleagues who are using it?


Carl:
It’s not been helpful to me yet. I've been at Cooper 20 years and I've never taken a sabbatical, but I will be applying to do one next spring because I want to get my hands dirty on this. I want to not just understand it on the theoretical side. I have tried to keep up with the reading and the paper, but I haven't used it much. PyTorch and TensorFlow are more common in deep learning. I've done independent studies with students on it. I've thrown together some layers and trained it on a small training set, but I want to get my hands dirty and do it at a lower level–to understand the lower level abilities of the library and try to create my own sorts of layers. So, I might have a different answer for that a year and a quarter from now.

Only the biggest companies in the world can train these LLMs from scratch. We can take pre-trained LLMs, maybe fine tune them and use them on data sets but that's another thing. I almost think the field is less interesting now because of how big these things are. Twenty plus years ago, I was a graduate student in NLP, and I was disappointed that the field was all on the machine learning side, and not on what I would call the computational linguistic side. Now I think linguistics is being influenced by NLP instead of the other way around. I don't find throwing a lot of data at a giant enrollment interesting. Is it what works? Yes–it's absolutely what works. Is it always going to be the thing that works best? Maybe. The way that I did things in grad school is no longer the way that it's done. I'm keeping up with it in terms of reading, but I want to be able to use it for research for colleagues.

There was recently a forum where faculty and students could talk about generative AI and the way that it's affecting classes and academia in general. And there were a couple of faculty members there who said they try to use it to help write the first draft of syllabi, and try to give projects that require students to use it. They didn’t have a particularly positive view of it but also not a fear of it. It was interesting to see. Some faculty seem intimidated by it. Some faculty think it’s great and that we have to use it in a positive way. There are colleagues who are using it and who are encouraging the school to use it. It came up at this forum to try to get to a place where faculty and students who want to can access the paid version of ChatGPT.  If I do this sabbatical I want to use GPT, but also implement my own systems with PyTorch. Both on local GPUs and on cloud-based platforms. I want to get my hands dirty with a lot of these things.


Kit:
I mean I know Gemini, you know what used to be Bard–Google's AI. They're working on rolling out a much larger effective memory. So that it's not like you're training the model yourself. Your interactions with it will be preserved over a much larger body of data. So over time, it certainly has a much stronger sense of the way that you're using it, what your knowledge is. I know this is sort of alongside open AI–these things that they call very confusingly GPTs.


Carl:
Yeah, I know.


Kit:
They use their GPTs and they, they sort of proprietary I think right like it's, yeah, yeah, you can build and you can build like, you know, your own little miniature version of that GPT that knows that you know you can feed your notes into it. The notion being partly that you get to a point where you're almost sort of chatting with yourself. Because it's built on the data that you have chosen to feed it. But that's fundamentally different from what you're talking about, in terms of genuinely getting your hands dirty.


Carl:
I want to at least have the experience and ability to create my own architectures–not just by stringing common layers together. What if I want to change the way the nodes work or create my own loss function? What if I want to have parallel paths in the architecture join in the way that I define? We would implement everything from scratch. I think with SVMs [Support Vector Machines], I used an existing library because it was too complicated, but every other machine learning method that I used as a grad student, I implemented from scratch. Even in a class, students use libraries and put together existing layers the way they want. That is something that's easy enough, but I want to experiment at the lower levels.


Kit:
Can you give some examples of the types of work you've asked students to perform with generative AI? This connects to something that I think has been emerging as we've talked: given the massiveness and expenses and computer power associated with the chat GPTs and the Geminis of the world, there’s tension between students and faculty really understanding these things and being able to play with them, versus this increasing concentration of power and resources far away from us. With that problem in the back of our minds, I'm curious how you work with students on this stuff.


Carl:
I think this is another thing that's going to have to grow going forward. We have a course in deep learning and it’s not all generative–it's also like categorization and training to classify. In one semester, there's only so much I can do even though it's a master's-level course that assumes backgrounds in things like DSA [Data Structure and Algorithms] and programming experience. We talk about conventional NLP and things like tokenization. We talk about computational linguistics and do a project based on that–they write up a parser for English based on a grammar. And then we get into deep learning and NLP which includes generative AI but also, you know, deep learning more generally. I give an open-ended project where they can install PyTorch or TensorFlow, follow tutorials, learn how to use it, then write up a report on it. That's as advanced as it can get in one semester unless I drop all the conventional stuff.

Many students have done independent study projects or master’s projects. I've had students do follow up ISs on advanced NLP where we spend a whole semester on a project using deep learning. Again, they're not all generative AI–the classification of text or images that's not generative, but relies on learning.

In class, it's smaller projects–they're open ended but I'll list a bunch of ideas that students can do and they'll pick one. They're not using things like chatGPT–they're using libraries and existing architectures (or at least existing layers), but at least they're going through a training and testing phase. Some of those projects have involved generative AI. Students train a natural language generation system on a corpus like Harry Potter or the works of the Wall Street Journal, and show that if you train it on this you get text that looks like this, and if you train it on that, you get text that looks like that.

I've struggled with what the next revamp of NLP will look like. It’s got to be two courses, but I don't have the time to do it as two separate courses.


Kit:
Do you worry that if you try to keep up with where the tech has gone these past two or three years, you would then leave out some root systems? Like by focusing on sort of the plant above the soil, you accidentally would then by not teaching people fundamentals? The field could accidentally cut off a whole new set of developments?
Carl Sable: The fundamentals are always crucial. I was surprised that on the news recently, the headline was that the CEO of Nvidia–I think his name is Wang– said that soon people aren't going to have to learn how to code anymore. A generation of code is going to get so good that you can just describe in English or your native language what you want a program to do and it'll be able to write it. I could be proven wrong, but I don't think it's true–I think it's not going to get to that point. Possibly ever. But even if the prediction is true, I would disagree that it's not important to understand how to code, because if you don't know how something works at a deep level, you're not going to use it appropriately. There was a period of time where calculators were popular, but that doesn't mean that students shouldn't learn arithmetic anymore. You have to learn how the low-level stuff works. You have to learn how to use the tool, but you also have to know how everything works. I would choose the fundamentals over the tools if I could only pick one, but, ultimately, students are going to have to learn both.