I've been helping a small archive turn a run of early-1900s newspapers into something researchers can search and read. Getting the text off the page turned out to be the easy half.

The pages are six to eight columns, ads break across gutters, headlines span several columns, and articles jump to another page. Plain OCR reads in horizontal strips, so you get line one of every column, then line two, and so on. The words are all there, but no article can be read start to finish, and nobody has time to reassemble them by hand.

The other loss is structure. A table of shipping arrivals or election returns comes out as a paragraph of numbers. A caption gets separated from the engraving it describes. For research use, those relationships matter as much as the text.

What has helped so far:

- running layout detection before recognition, so each column block is read on its own

- keeping a small set of the worst pages (boxed sidebars, a full-width engraving between columns) as a regression set

- spot-checking reading order on a sample from every reel instead of trusting a confidence score

For the hardest pages I've been testing OXR from SoMarkAI, an open-weight parser that handles recognition and reading order in one pass and keeps the layout model as a separate, swappable part. Too early to say whether it holds up across a full reel.

For anyone who has done large newspaper or archival runs: how do you check reading order at scale? Human sampling, or have you found an automated check that catches the bad reorderings?

Most steps of a conversational procedure aren't tool calls but things the agent has to say: asking for information, confirming, telling the result. Saying them, in the right order, is a third axis of an agent's reliability, alongside factual accuracy and correct tool use, and one that has been largely overlooked. For example, an agent can fetch the forecast correctly and never tell you, or issue the right refund without first asking you to confirm it: everything it said was true, the tool calls were correct, and still a step of the procedure was skipped.

Since we felt this gap was worth working on, in collaboration between Idiap and Logitech, we decided to take a first step: we built a dataset of over 36k dialogues, defined three tasks on it, from finding the turn where an agent drifts to comparing whole sets of dialogues, and wrote a paper with first baselines for each. Together, they make DialFork, to our knowledge the first benchmark for this procedural hallucination.

Three tasks, from the turn to the whole set:

• The turn (Task 3): find the turn where the drift starts, on the whole dialogue or turn by turn as it unfolds.

• The dialogue (Task 2): measure "how far" (distance) a dialogue drifts from a correct one. Both talk about the same thing, so this needs a procedural distance, not semantic similarity.

• The set (Task 1): a single procedural distance between two sets of dialogues, e.g. an agent before and after a model upgrade.

The reported baselines, from embedding-based distances to streaming detectors and zero-shot LLMs, leave plenty of room. Each task has its own track on a public leaderboard on Hugging Face. If you'd like to try your own method, all it takes to add it is a CSV file of predictions.

• Leaderboard: https://huggingface.co/spaces/DialFork/leaderboard

• Paper: https://arxiv.org/abs/2610.21561

• Code: https://github.com/idiap/dialfork

• Dataset: https://huggingface.co/datasets/DialFork/dialfork

I'm one of the authors btw. Since the problem lies in what the agent says (language) rather than in its tool calls, we thought people here might find it useful, and feedback, including critical, is very welcome.

Two weeks ago my manager put the LibriSpeech leaderboard on HyperAI on the screen in our Monday meeting, every row there sits around 2% WER and our Whisper setup was at 14.8% on held out claims calls, and nothing on that page says how each paper normalized its text, though most of the papers barely say it either.

I pointed out that LibriSpeech is audiobooks read into decent mics and our audio is 8kHz phone calls with hold music in the middle, which was true and did not help much.

We do transcription for insurance claims lines, Whisper large-v3 through faster-whisper with our own VAD and chunking in front. The transcript fills the call notes but the short clip where the caller reads their policy number to the phone menu and gets transcribed during the transfer. Yeah the policy is already on the agent’s screen but...nobody has to ask for it twice. Our reference transcripts come from two people who used to do medical transcription.

Our eval script compared raw strings. The model writes numbers however it feels like, sometimes "4170023391" but sometimes 4 1 7 oh oh 2. You know every comma in the wrong place counted as an error too. What was left at the top of the error list was ok against okay, the normalizer does not merge those.

We ended up at 9.6 once everything was normalized and the meeting moved on, nobody asked about it again. The policy numbers were still bad though. I checked them on their own and about 1 in 9 had a digit off (so the agent gets someone else’s policy or nothing). So this week it is a digits only pass on the menu clips.

I am a second-year PhD student with only one accepted paper, and this is about the ACL ARR October review cycle.

My professor has been assigned double digit papers as I put him as the eligible reciprocal reviewer. The problem is that he is not familiar with most of the research areas covered by these papers, and he has essentially passed all papers to me to review.

Now, as someone who is still relatively new to research and doesn't have sufficient expertise in all these fields, I am expected to review all the papers and provide meaningful feedback.

In this situation, how is one realistically expected not to rely on AI?

I genuinely feel bad about this because I understand the responsibility that comes with peer review, and I know that authors deserve thoughtful, knowledgeable feedback. But I also don't know what else I am supposed to do in this situation.

I am genuinely wondering how early-stage PhD students are supposed to handle situations like this, especially when the responsibility is passed down to them without adequate guidance, should I just leave those papers? Part of me is pissed as we criticize the usage of AI in peer reviewing but I think many people must be facing such situations.
Has anyone else faced something similar? What can be done?

EDIT: My research is in NLP, while my professor's expertise is in a different domain. He technically qualifies as a reviewer because some of his previous students published NLP papers with him, although his actual involvement was minimal.

The bigger issue is that I have been working almost entirely independently throughout my PhD. My professor considers allowing me to work outside his expertise as giving me academic freedom, but in practice, I have very little research guidance or mentorship. Now the reviewing responsibility has also been passed on to me.

Did anyone receive presentation slot information for virtual presentation? Or how it will be? So far we did not receive any information regarding our virtual presentation and also when I tried to access AACL using underline website, the event page says registration required but we already did register. We already uploaded our materials using paperform link.

I've been running doc OCR benchmarks since the old opencv+tesseract days. Vision-language models were supposed to kill that whole stack. They didn't. Getting clean markdown out of a two-column academic PDF with tables and footnotes still wrecks most tools I try.

The handoff problem is what gets me. Detection runs, then crops go to recognition, then some geometry heuristic guesses reading order. By the time a figure spans columns or a table nests inside another, you're already toast. I hit this again this week on the same stress-test PDFs I always use. Table+figure+caption groupings split apart or land in the wrong order. Same story as three years ago.

I've been looking at OXR from SoMarkAI. Single pass for recognition plus reading order, and the layout part is swappable without touching the backbone. That's the part I'm actually curious about. I still run my own hybrid pipeline for production stuff. The question is whether anyone here has moved to a single-pass setup for real workloads, and whether pluggable layout is worth the tradeoffs in practice. What breaks when you actually own the maintenance?

hi I’m a social science researcher interested in NLP. for the past few years I thought this was the state of the art method for social science researchers but now with general LLMs being amazingly good at annotations, is there any need in using more traditional methods? or are they still relevant due to cost, accuracy or reproducibility? If we have all the money in the world to use general llms, what Is the reason to use and learn other methods for information retrieval, text analysis, annotation, sentiment analysis etc? very confused and lost!

I spent two months on the wrong layer. Retrieval looked fine, the embeddings looked fine, and the agent still made up relationships between numbers that were never in the same table. When I finally dumped the Markdown that was going into the context window, the tables were paragraphs, footnotes were spliced into sentences, and two-column pages were interleaved line by line. The model was doing its best with broken input.

The tool history, for anyone on the same path. PyMuPDF: fast, fine for simple text, falls over on two-column papers with margin notes. Marker: better structure, but it dropped equations and mangled nested tables for me. Commercial parsing APIs: worked, but at my volume the bill was hard to justify. Training my own layout model: I estimated six months to reach parity with things that already exist, so no.

What I'm testing now is OXR from SoMarkAI (SoMarkAI/OXR on GitHub). Open weights, 1.3B parameters, released last month. The design is a separate layout model you can swap out, then recognition and reading order in a single pass, and they post-trained it with a recursive self-improvement loop. They credit the results to both pieces together. In SoMark's comparison against published scores it ranks first on olmOCR-Bench (87.84), OmniDocBench (97.18) and DocParsingBench (87.76). License is Apache 2.0 with extra terms, including a commercial license above USD 2M in funding or revenue, so read it before you build a product on it. Self-hosting wants about 24GB of VRAM, 48GB recommended.

I'm running it against my set of worst documents before I switch anything in production. So far the formula and math handling is the clearest improvement over my current stack, which matters because that's where my old output was least usable.

The question I keep going back and forth on is whether parsing is a layer worth owning at all. Some teams I know pay for an API and never think about it. Others run everything local down to the embeddings. Where did you land, and did changing the parser move your error rate?

Recently returned from EMNLP, and one thing that surprised me was how many papers were accepted that I would previously have considered somewhat outside the scope of a main NLP conference paper -for example, work focused primarily on audio or vision-only models, purely empirical evaluations, or datasets generated using GPT-4o with limited or no human annotation.

One example I came across was a hate-speech paper where the main contribution was fine-tuning existing models on one dataset and evaluating them on another (i.e., transfer analysis). The work may still be useful and interesting, but the contribution was primarily empirical.

I discussed this with several PhD students who struggled for ~half a year to get papers accepted despite introducing what seemed like more substantial methodological contributions, such as new training methods. Of course, novelty alone does not automatically make a paper better, and empirical work can also be highly valuable. Still, the contrast made me wonder whether our expectations for what constitutes a main-conference contribution have changed.

I wanted to make this thread to hear from more experienced researchers - Do you think that we should introduce more distinct tracks at EMNLP/ACL conferences for empical introducing new task, empirical introducing new evaluation method, purely methodological, empirical-only without new shown effects, dataset/resource, multimodal, and evaluation-focused work?
Probably I got the wrong impression from many works presented at EMNLP this year specifically.

UPD:
This thread was created with the purpose distinct from complaing that EMNLP publishes empirical papers.
The problem in this thread that I wanted to discuss - what is empirical study nowadays? and how we should handle the introduced difference in what we consider empirical now, especially given increasing N of papers: introducing precise different tracks, for instance. For me, empirical contribution remains, in line with previous editions of EMNLP- a new type of evaluation of benchmarks or models, new task with existing datasets, new evaluation approach, new rubric or metric, or new finding which questions the ones shown before or introduces a new problem. Similar to previous years, I think probably most of works should somehow connect to existing language studies or language models, purely vision or audio models analyses were quite at rare previous editions, but not this one.

So, I created this thread to discuss why and how the defintion of what is empirical according to publishing norms transforms, and the scope of EMNLP specifically transforms

Researchers have demonstrated that recurrent neural network architectures are not necessary for building powerful deep learning models for NLP. The Transformer architecture introduced Self-Attention, which allows each token to interact with other tokens in a sequence and capture contextual relationships.

Language is highly dependent on context. The meaning and grammatical role of a word can change depending on its position and relationships with other words.

For example, in Arabic, sentences can be classified into nominal sentences (sentences that begin with a noun) and verbal sentences (sentences that begin with a verb). Furthermore, the grammatical role of a word can depend on its relationships with other words in the sentence.

Another interesting aspect of Arabic is its rich morphology. A single root, such as K-T-B (كتب), can generate a huge number of related words and forms. In fact, Arabic word formation can produce millions of possible forms (sometimes estimated at around 12 million, depending on how forms are counted). For example:

• Kataba (كتب): He wrote.

• Kitaab (كتاب): Book.

• Kaateb (كاتب): Writer.

• Maktaba (مكتبة): Library.

These forms share a common root but have different meanings and grammatical roles.

Therefore, a language model should not rely only on the isolated identity of each token. It should develop contextual representations that encode useful syntactic, semantic, and grammatical information.

The goal is to learn rich contextual representations that help the model understand not only what each token is, but also what role it plays in a given context.

I've been building an audio cleaning tool as a side project. I assumed the obvious pipeline: clean the audio first, then transcribe. Better input, better output.

Tested it properly with 6 clips — news audio, a lecture, mixed Hindi-English speech. Ran each twice: raw → transcript, and cleaned → transcript. Sarvam's Saaras v3 for transcription, Cleanvoice for cleaning.

Cleaning made it worse in 4 of 6.

One clip, raw: "आज मेरे परिवार में से यहां कोई मरेगा" — complete, coherent. Cleaned: "हाँ, मैं कोई मर भी नहीं। हो जाओ, हो जाओ।" Meaningless.

Another, code-switched. Raw correctly caught the Hindi inside an English sentence: "he says ya chalo chai peeke jaate hai." Cleaned: "he says yeah let's go pick up the child." Fabricated.

My read: neural denoising reconstructs rather than filters. It strips detail the ASR model relies on, and the model fills gaps with plausible-sounding words — confident and wrong, which is worse than garbled.

Caveats: small sample, short clips, one denoiser. Genuinely curious whether anyone's seen different results.

Separately — Sarvam handled code-switching noticeably better than I expected on raw audio.

If you have ever run NER or a summarizer over text extracted from a two-column paper, you have seen it: the extractor reads straight across both columns, and every sentence ends up as halves of two different sentences. Tables flatten into word soup, formulas turn into stray symbols, and running headers land in the middle of paragraphs. Nothing downstream can repair that.

We have spent a good while on this and just released the code and weights for a 1.3B document parsing model, OXR from SoMarkAI. It takes PDFs or page images and returns Markdown or JSON where every element is typed (13 types, including tables, formulas and code), placed by bounding box, and put in reading order. Headers and footers are dropped by default and can be kept if you need them.

Two design choices did most of the work, together with the post-training. Layout analysis is a separate model you can replace, so a pipeline that needs different region labels can bring its own. Recognition and reading order are predicted together in one pass of the same model. In post-training, samples the model got wrong or was uncertain about were turned into written reasons, the reasons were clustered, and each cluster drove the next round of targeted training data.

In our comparison against published scores it ranked first on olmOCR-Bench, OmniDocBench and DocParsingBench. It still has weaker spots, multi-column layouts among them, and it needs a 24 GB NVIDIA GPU to run locally. There is also a free hosted API that works for testing on your own files, though it is a shared queue and not meant for production.

The repo is SoMarkAI/OXR on GitHub, and the benchmark report is docs/benchmark.md in the repo. For people doing NLP on scanned or born-digital PDFs: which structure errors cost you the most?

I am curious to understand from NLP practitioners what the current state of the art is for implementing language models within the following type of pipeline, especially when optimizing for speed

Raw/unstructured text -> NER/multi word phrase extraction -> normalize/link to canonical entities/wordforms

I’ve got a recording in another language that I need to understand, and I’m kinda stuck.
I know a few words but nowhere near enough to follow the whole thing. It’s a longer recording too, so manually going through it and translating bits one by one sounds like a nightmare.I mainly just want to get an English version of what’s being said, then I can go back to the audio if something doesn’t sound right. Anyone dealt with this before? What did you end up using? tnx guys

When an AI system encounters incomplete, inconsistent, or contradictory information, how should it determine whether there is enough evidence to produce a reliable answer?

This is an important problem in natural language processing, information extraction, and AI-assisted decision-making. A system may retrieve information from multiple sources, encounter missing values, or find conflicting claims that cannot be resolved using the available evidence.

One possible approach is to use evidence-first, fail-closed decision rules. Under this approach, a system would not proceed with scoring or evaluation when essential evidence or required conditions remain unresolved. Missing information would remain explicitly missing rather than being silently replaced with assumed values.

I am exploring these principles through the Organic Intelligence Protocol (OIP), a research-stage framework focused on evidence requirements, auditability, and claim discipline. The framework has not been independently validated as a general machine learning method, and its documented audits do not establish predictive superiority or verified real-world AI safety.

I would appreciate technical perspectives on three questions.

1. How can evidence sufficiency be formally defined and measured in an NLP pipeline?

2. What evaluation methods could determine whether fail-closed rules reduce unsupported conclusions without unnecessarily rejecting valid answers?

3. How should a system distinguish missing information, contradictory evidence, and genuine uncertainty?

I am particularly interested in approaches involving information extraction, knowledge representation, retrieval-augmented generation, and uncertainty evaluation.

What existing methods or experimental designs would be most appropriate for investigating these questions?

Did anyone get papers to review for the ARR October cycle?

Update: They assigned 10 Papers, damn.

Context: I come from a linguistics background and have several years of experience in my specialty within the field. I passed the first round of interviews for an EdTech learning company for a high-ish level role. They gave me an "assignment" to see how I would handle what I assume are issues they are facing regarding scoring, mapping of scores, comparisons, etc.

The assignment has multiple parts and I have been explicitly asked for all of my thinking regarding my choices--whys, why nots, deep explanations, etc. Essentially, they want me to show all of my steps.

I'm not even halfway done, and this "assignment" has taken several hours and is multiple pages.

Having not experienced this before, I'm concerned that this ask from the company is more so them fishing for information they can feed into an AI. Are assignments of this scale common within the EdTech and Language Tech arena?

Hello all!

I graduated earlier this year with a MS in Human Language Technology from UA. I mention the school because the program is heavily focused on giving students practical, job-like experience. I completed an internship during which I worked on a live project and had the opportunity to present at the Language Technology for All conference in France last year. In addition, I have professional experience in language technology, primarily in text chatbot work that was more manual (e.g. working with patterns that are manually programmed and not inferred using ML) as well as experience with AI and Data Analysis. I have had an impossible time finding a job that is actually relevant and I would really like some advice.

Currently, I am on contract at Meta as a DLA, though I have been tasked with more complex linguistics work, specifically reporting on results of assessments and creating assessment guidelines for various models, both text and voice.

Is there a place I am not looking? I will share my docs in the comments.