Quick Answer: The ctresb/leis project provides a complete brazilian laws markdown dataset containing over 323,800 federal, state, and municipal statutes. Converted from official sources into clean Markdown with YAML front matter and URN Lex identifiers, it enables engineers to train LLMs, build RAG pipelines, and audit legal revisions directly through Git version control without writing brittle web scrapers.

Scraping legal text directly from government websites like Planalto or state legislative assembly gazettes usually ends in broken DOM parsers, corrupted character encodings, and missing historical revisions. The ctresb/leis repository changes this workflow by compiling a standardized brazilian laws markdown dataset housing over 323,000 statutes. By converting unformatted PDFs and legacy HTML into plain text with structured metadata, this open repository gives software engineers, legal tech founders, and NLP researchers direct, programmatic access to Brazil's entire legislative history through standard file trees.

Building legal AI tools in Brazil traditionally required maintainers to build dozens of custom scrapers. Every state legislative assembly (such as ALESP in São Paulo or ALERJ in Rio de Janeiro) uses different portal software, custom sessions, and inconsistent URL patterns. Federal portals suffer from HTML nested tables created in the late 1990s, where inline styling sits alongside inconsistent ISO-8859-1 and UTF-8 character encodings.

When feeding raw web content into Large Language Model (LLM) tokenizers or vector databases, these encoding quirks introduce noise. A chunker encounters missing paragraph tags or arbitrary table borders, fragmenting articles across vector boundaries. The popular advice suggests writing headless browser scripts using Playwright or Puppeteer to render dynamic pages before parsing DOM trees. That advice falls short because official portals often change their frontend DOM structures without notice, breaking scraper pipelines silently.

Official Portals (PDFs/Legacy HTML)
       │
       ▼  (Brittle Custom Scrapers)
   Raw Files (Inconsistent Encodings, Missing Tags)
       │
       ▼  (Fragmented Token Boundaries)
  Poor Embedding Quality & Broken Retrieval

Consuming a pre-built, standardized Markdown legal corpus eliminates this maintenance headache. The ctresb/leis repository normalizes every law into canonical document types (lei-ordinaria, lei-complementar, decreto-lei), stripping away interface layout bloat while retaining exact textual hierarchy.

That said, raw text alone isn't enough until you can query its metadata cleanly...

Anatomy of the Brazilian Laws Markdown Dataset: Front Matter and URN Lex

Every file in the repository (outside the root constituicao.md) opens with a standardized YAML front matter header. This design choice separates document attributes from the body text, allowing static site generators, search indexes, and parsing scripts to read metadata without scanning millions of tokens.

Here is an example YAML header from a processed statute:

---
titulo: "Lei nº 10.406 de 10/01/2002"
uf: "SP"
tipo: "Lei Ordinária"
numero: "10406"
data: 2002-01-10
ano: "2002"
ementa: "Institui o Código Civil."
situacao: "Em vigor"
revogada: false
versao_texto: "compilada (texto vigente)"
publicacao: "DOU de 11/01/2002"
data_publicacao: 2002-01-11
urn: "urn:lex:br:federal:lei:2002-01-10;10406"
fonte: "https://normas.leg.br/?urn=urn:lex:br:federal:lei:2002-01-10;10406"
baixado_em: 2026-10-13
---

Notice the urn key. It implements the URN Lex standard (urn:lex:br:...), a persistent uniform resource name identifier used across Latin American legislative systems. By standardizing URN references, systems can resolve cross-statute references automatically without relying on brittle relative URLs or platform-specific primary keys.

Directory organization mirrors political geography and regulatory hierarchy:

  • federal/: Acts from 1822 to present divided by year and type.
  • estaduais/UF/: State-level legislation grouped by Brazilian federative unit.
  • reguladores/<orgao>/: Infralegal acts from regulatory agencies like ANVISA, CONTRAN, and RFB.
  • jurisprudencia/<tribunal>/: Binding court precedents, constitutional controls, and court summaries.
  • municipais/UF/<municipio>/: Municipal municipal legal frameworks.

Here's where it gets interesting when you look under the hood of Git's commit graph...

Git as a Database: 323,000 Commits for Historical Auditing

Rather than dumping thousands of files in a single initial commit, the repository creators constructed a git graph containing over 323,800 historical commits using git fast-import. Each initial commit represents a single statute, backdated to the exact day it was promulgated and credited to the ruler or official who signed it.

According to analytics generated from scripts/estatisticas.py, over 5,414 unique historical signatories appear in the commit log. Prominent figures dominate the sign-off history:

  • Marconi Ferreira Perillo Júnior: 4,247 signed statutes
  • Aécio Neves: 3,795 signed statutes
  • Geraldo Alckmin: 3,457 signed statutes
  • Ronaldo Caiado: 3,097 signed statutes
  • Getúlio Vargas: 2,654 signed statutes

Working with pre-1970 dates presents a real technical edge case in software engineering. Because Git internally tracks author timestamps as Unix epoch offsets (seconds elapsed since January 1, 1970), dates prior to 1970 use negative integer offsets. The build scripts (scripts/montar_repositorio.py) handle these negative timestamps explicitly, preserving accurate timeline ordering back to imperial statutes enacted in 1822.

This structure creates a massive architectural shift compared to standard relational databases...

When architecting legal tech software, choosing how to store and query statutory text dictates system performance and ongoing maintenance cost. Below is a breakdown comparing the Git-backed open source brazilian legal data structure against traditional relational databases and dedicated vector indexes.

Feature / Metricctresb/leis (Git/Markdown)Relational Database (PostgreSQL)Vector Database (Pinecone / Qdrant)
Version HistoryBuilt-in native line diffing via GitRequires custom audit triggers or temporal tablesNon-existent (requires external metadata mapping)
Storage OverheadLow (plain text + compression)Medium (B-Trees, indices, storage bloat)High (dense vector embeddings and index memory)
Parsing VelocityDirect file read / local disk speedSQL query parsing & network latencyK-NN vector search execution time
Chunking UtilityDirect heading-based text splittersNeeds custom text unescaping scriptsPre-chunked prior to storage

For local development, pulling a static git repository eliminates API rate limits and vendor lock-in. Developers can grep through text locally or sync changes through incremental commits without running complex DB migrations.

This brings up a key operational question for developers building search engines...

If you want to know how to build legal RAG applications using a git versioned legislation corpus in plain text, the structure of Markdown simplifies the chunking phase dramatically. Standard document chunking strategies rely on arbitrary character counts (e.g., 500 tokens with 50-token overlap). This approach frequently cuts through legal articles, isolating paragraph conditions from their parent root article (Artigo).

Because the corpus normalizes heading structure, developers can construct custom section-aware chunkers. Follow these steps to process the files:

  1. Parse Front Matter: Read the YAML header using python-frontmatter to extract urn, situacao, and data into metadata attributes.
  2. Split on Headers: Parse headings (#, ##, ###) corresponding to Articles (Art. 1º), Paragraphs (§), and Items (I, II, III).
  3. Inject Scope into Metadata: Append the parent article title into every chunk's context payload. This prevents vector retrieval drift when an isolated paragraph simply states "Revogado pelo Decreto nº...".
  4. Embed Metadata and Body: Pass both the statute metadata (UF, year, status) and structural chunk to your embedding model (e.g., text-embedding-3-large).

By leveraging this structure, vector search queries return exact article context rather than truncated fragments.

Most developers assume OCR output from public archives is immediately production-ready — don't make that mistake...

Data Quality Failure Modes: Handling Scanned PDFs and Missing Signatures

While the dataset is extensive, working with historical public records introduces data quality limits that every engineer must handle. The repository relies on automated conversion pipelines and OCR (Optical Character Recognition) applied to scanned PDFs downloaded from state gazettes.

In roughly 48% of historical state statutes, the signature block was either unreadable by OCR or absent from the underlying PDF image layer. In these scenarios, the script scripts/signatarios.py assigns institutional authorship (e.g., "Estado do Pará" or "República Federativa do Brasil") to maintain identity consistency across the git timeline. Synthetic email addresses (name-surname-NNNNN@exemplo.com) are generated deterministically based on string hashes to ensure identical authors collapse to a single ID.

Additionally, non-textual elements or corrupted scans are tracked in _falhas.jsonl rather than failing the build silently. The repository also retains 27 GB of raw HTML and compressed PDF originals under _fonte/ (excluded from git versioning) to allow re-processing when OCR tools improve.

Keep in mind: these converted text files carry an informative legal status and do not legally replace official publication in the Diário Oficial. Always verify the fonte URL listed in the front matter before submitting legal briefs in court.

Frequently Asked Questions

What is the primary source of the brazilian laws markdown dataset?

The dataset extracts legal texts directly from official federal, state, and municipal sources, including the Planalto administrative database, LexML repositories, state legislative assemblies, and official gazettes. Each Markdown file header contains a fonte field pointing directly to the original source document.

Parse the YAML front matter to extract statute metadata, then use a Markdown-aware text splitter to divide articles into chunks along heading lines. Index the resulting chunks into a vector database alongside metadata fields (uf, ano, urn) to allow metadata filtering during semantic retrieval.

Are these Markdown files legally binding in Brazilian courts?

No. The files provided in the repository serve an informative purpose and do not replace official publication in the Diário Oficial. For official legal representation or judicial compliance, always confirm the text against the source link provided in the document header.

How does the repository track incremental law updates?

After establishing the initial 323,000-commit historical baseline, new legislative updates are ingested via batch commits per collection run. This avoids re-writing thousands of historical git SHAs while keeping the active codebase synchronized with current law.

Using a clean, structured brazilian laws markdown dataset eliminates months of tedious web scraper development and text cleaning. By combining standardized YAML metadata with native Markdown structural headers, the ctresb/leis repository provides the foundational layer for high-precision legal search engines and AI agents.

Clone the repository locally this week, run a basic chunking script over the federal/leis-ordinarias/ directory, and test your retrieval accuracy against raw PDF extractions. Pass this resource to someone wrestling with legal data extraction, and read our breakdown on building vector indexes for legal documents next.