Indian Scholar, Dr. Rishi Rajpopat, at Cambridge solves a 2500-Year-Old Sanskrit Algorithm Problem in Panini’s Texts

Rishi Atul Rajpopat, a Ph.D. scholar at the faculty of Asian and Middle Eastern Studies at St. John’s College, Cambridge, has solved a grammatical problem posed by the texts written by ancient Sanskrit scholar Paṇini. While researching for his Ph.D. thesis, published on 15th December 2022, Dr. Rajpopat decoded a 2500-year-old algorithm that makes it possible, for the first time, to accurately use Pāṇini’s ‘language machine’

In his thesis titled “In Panini, ‘We Trust: Discovering The Algorithm For Rule Conflict Resolution In The Astadhyayi” Dr Rajpopat solved the grammatical conundrum bugged scholars for two millenia. For months the solution evaded the Indian scholar as well, as he tried to decode Panini’s Language Machine. ‘Rule Conflict’ has been the root of all confusion and conflict as two or more Panini’s rules are simultaneously applicable at the same step. How would you even choose which one to use? Solving this ‘Rule Conflict’ that affects a legion of Sanskrit words like Guru and Mantra required an algorithm. After months of pondering over the problem without a breakthrough, Rajpopat took a brief sabbatical before furiously getting back to work. Earlier his Sanskrit Professor and Ph.D. advisor Professor Vincenzo Vergiani remarked that if the solution is complicated then you might be wrong. This struck a chord with Rajpopat. It was an Eureka moment, states Rajopat when he made the breakthrough. Computer scientists had already given up over 50 years ago working on a rules-based approach in Natural Language Processing. Decoding Panini’s Machine would enable machines to learn grammar-based Sanskrit to produce human speech. Rajpopat believes this will be a milestone in the history of human-machine interaction and India’s intellectual history

Paṇini was a sanskrit grammarian who gave a comprehensive and scientific theory of phonetics, phonology, and morphology. Sanskrit was the classical literary language of the Indian Hindus and Panini is considered the founder of the language and literature.
In the Aṣṭādhyāyī, he outlines rules to produce variations of a root word in such a way that they are grammatically and syntactically correct according to the rules of Sanskrit. These include grammatical rules for the formation of new words, such as sandhi or joining two words to produce a third.
About 4,000 rules are compiled in the text as sutras, and each book or segment outlines a step-by-step method – like an algorithm checking for conditions – to produce and construct new words. But the sutras are not always obvious in their meaning. Since they are concise and made up of limited words, they can be confusing to modern readers.
Additionally, these sutras are not stand-alone – each is built upon the preceding set of rules in a linguistic device called anuvr̥tti, or the continuation of one rule into the next. This means that sutras use recall and reference keywords that were used previously.

Paṇini’s text Aṣṭādhyāyī, which comprises a set of rules to derive or form new words from root words, often contains conflicting rules for creating new words, with many scholars confused about which rules to use. Solving such conflicts in this linguistic algorithm of a book was the subject of many scholars’ interests.

Paṇini wrote a meta-rule to resolve rule conflicts, which so far scholars have interpreted as: In the event of a conflict between two rules of equal strength, the rule that comes later in the grammar’s serial order wins. In his dissertation, Rajpopat argues that this metarule was historically misunderstood – instead, Paṇini meant that between rules applicable to the left and right sides of a word, he wanted the reader to choose the rule applicable to the right side.
With this logic, Rajpopat finds that Paṇini’s algorithms produce grammatically correct words and sentences without errors.

Generally, scholars have interpreted Pāṇini’s meta-rule as meaning. In the event of a conflict between two rules of equal strength, the rule that comes later in the grammar’s serial order wins. Rajpopat rejects this, arguing instead that Pāṇini meant that between rules applicable to the left and right sides of a word respectively, Pāṇini wanted us to choose the rule applicable to the right side. Employing this interpretation, Rajpopat found Pāṇini’s language machine produced grammatically correct words with almost no exceptions. He explains it by taking ‘mantra’ and ‘guru’ as examples. In the sentence ‘devāḥ prasannāḥ mantraiḥ’ (‘The Gods (devāḥ) are pleased (prasannāḥ) by the mantras (mantraiḥ)’ we encounter ‘rule conflict’ when deriving mantraiḥ ‘by the mantras’. The derivation starts with ‘mantra + bhis’. One rule is applicable to the left part ‘mantra’ and the other to the right part ‘bhis’. We must pick the rule applicable to the right part ‘bhis’, which gives us the correct form ‘mantraiḥ’. And in the sentence ‘jñānaṁ dīyate guruṇā’ (‘Knowledge [jñānaṁ] is given [dīyate] by the guru [guruṇā]’) we encounter rule conflict when deriving guruṇā ‘by the guru’. The derivation starts with ‘guru + ā’. One rule is applicable to the left part ‘guru’ and the other to the right part ‘ā’. We must pick the rule applicable to the right part ‘ā’, which gives us the correct form ‘guruṇā’.

Sometimes, it is also confusing whether a sutra references any previous rules, and sometimes, it is unclear whether a rule needs to be continued to the next. More often than not, even when a rule is understood in its meaning, it is unclear where to apply it.
Perhaps most confusingly, when a linguist knows where to apply one rule, another conflicting one also becomes applicable at the same step. This leads to one rule blocking the other, or both blocking each other. This rule conflict is where Rajpopat’s thesis comes into the picture.
As an example, Rajpopat mentions the words formed by the root vr̥kṣa. Combining the words vr̥kṣa and bhyām produce vr̥kṣābhyām, where the last ‘a’ sound of the first word is replaced by a longer ‘a’ sound. On the other hand, combining vr̥kṣa and su gives vr̥kṣeṣu where the same sound is replaced with ‘e’. Now, when the word vr̥kṣa has to be added to the plural bhyas, should the ‘a’ be replaced with a longer ‘a’ or ‘e’? Various scholars had argued that different rules were to be interpreted in different ways, but Rajpopat’s Occam’s Razor solution gives the correct form, vr̥kṣebhyaḥ. Rajpopat’s work solves simpler conflicts, as illustrated by the guru example above, and more complex ones. To illustrate the more complex rule conflict that Rajpopat solved with an example, consider the word for “from god” which are split into their roots deva + bhis.
When these two words are to be joined together, there are two rules that conflict: Rule number 7.3.103 applies to the word ‘deva‘ and states that when the words are joined together, the ‘a’ should become ‘e’, to form devebhih. Rule number 7.1.9 applies to the word bhis and states that the ‘a’ sound preceding bhis should be replaced with ‘ais‘, making the new word devaiḥ. Under such a circumstance, previous scholars have interpreted the metarule to mean the later sequential rule, ie., 7.3.103. But devebhih is not the correct word, devaiḥ is. Scholars previously then interpreted this conflict as being an exception — in cases where the applicable rule provides a wrong result, the other rule needs to be applied. Panini’s meta-rule states vipratiṣedhe paraṁ kāryam, which traditional scholars have interpreted as ‘in the event of a conflict between two rules of equal strength, the rule that comes later in the serial order of the Aṣṭādhyāyī, wins.’
Rajpopat went back to the canon text and reinterpreted the meaning of the word “para” to ‘right-hand side’. By his interpretation, the rule 7.1.9 applies, making the new and correct word, devaiḥ.
This solution does away with a number of creative and complex linguistic workarounds that others have proposed in the past to resolve such rule conflicts. In a statement, Rajpopat’s supervisor Vincenzo Vergiani said, “My student Rishi has cracked it — he has found an extraordinarily elegant solution to a problem which has perplexed scholars for centuries. This discovery will revolutionize the study of Sanskrit at a time when interest in the language is on the rise.” Rajpopat recalled how he managed to arrive at his conclusion. “I had a eureka moment in Cambridge. After 9 months of trying to crack this problem, I was almost ready to quit, I was getting nowhere. So I closed the books for a month and just enjoyed the summer, swimming, cycling, cooking, praying, and meditating. Then, begrudgingly, I went back to work and as I turned the pages, these patterns started emerging, and it all started to make sense. There was a lot more work to do but I’d found the biggest part of the puzzle,” Rajpopat said in the statement. He worked for another two and a half years on the project before finishing his thesis.

Sanskrit Language for Computers

While Sanskrit is often regarded as a “language suitable for computers”, it’s just a language like any other.
However, Paṇini’s comprehensive texts could prove useful for natural language processing (NLP), where computers contextually understand natural language and its nuances. Machine learning algorithms struggle with aspects of NLP such as natural language understanding and generation. A rule-based approach, such as that of Aṣṭādhyāyī, can find potential use in training AI systems for natural language processing (NLP).
Paṇini’s work is not the first one to delve into detailed grammatical rules, but it builds on previous work. His treatise is one of the oldest texts written in and about the Sanskrit language that has survived entirely and is also linguistically one of the oldest in history. It was written in the Classical Sanskrit form, and in fact, marks the beginning of the Classical Sanskrit era.
Vedic Sanskrit preceded Classical Sanskrit, and texts written in Vedic Sanskrit were accompanied by Prātiśākhyas, which explained how to pronounce the texts in the Vedas. These supplementary texts were then expanded upon by scholars, eventually leading to writing comprehensive texts about grammar, Pāṇini writing his masterpiece, and then the interpretation of Pāṇini’s work, as done by Katyana and Patañjali (one of several ancient scholars of the same name), in the 2nd century BCE.
Sanskrit, which is a combination of the words sáṃ (‘together, good, well, perfected’) and kṛta– (‘made, formed, work’), was comprehensively edited by scholars and refined as a language when naturally evolving languages or Prakritic ones were being still spoken. As a result, the language consists of well-laid-out grammatical rules, which were first compiled by Paṇini.
Even today, Aṣṭādhyāyī stands out as a comprehensive and complete work that algorithmically describes a language and how to form new words in the language without misinterpretation. This lends itself to be machine-like in nature and, thus, is useful for training NLP models.
“Computer scientists working on natural language processing gave up on rule-based approaches over 50 years ago,” said Rajpopat. “So teaching computers how to combine the speaker’s intention with Paṇini’s rule-based grammar to produce human speech will be a major milestone in the history of human interaction with machines, as well as in India’s intellectual history.”

Reference
R.A. Rajpopat, ‘In Pāṇini We Trust: Discovering the Algorithm for Rule Conflict Resolution in the Aṣṭādhyāyī’, Ph.D. thesis (University of Cambridge, 2022). DOI: 10.17863/CAM.80099
https://www.cam.ac.uk/stories/solving-grammars-greatest-puzzle

Some Text by Tony Rai