repowisedatazip-inc/olake

Architecture of datazip-inc/olake

datazip-inc/olake is 374 source files holding 3,058 symbols across 10 modules, written mainly in Go, Java, Shell. The largest module is drivers, with 100 files, and 8 files are entry points into the dependency graph.

How olake fits together

Each source has its own connector directory under drivers, such as drivers/mysql/main.go, which serves as that connector's entry point. The main.go file in each driver relies on the single public entry in the root module to set up the shared protocol machinery and start the connector run. The files protocol/root.go, protocol/discover.go, protocol/sync.go, protocol/clear.go and protocol/check.go define the lifecycle operations, such as discover, check, sync and clear. The destination directory defines the writer interfaces and destination error handling, and destination/iceberg/iceberg.go and destination/parquet/parquet.go implement them for Iceberg and plain Parquet. The Iceberg writer also has a Java project in destination/iceberg/olake-iceberg-java-writer.

Largest modules

  • drivers: 100 files and 17,875 lines of code
  • destination: 55 files and 9,415 lines of code
  • utils: 35 files and 9,994 lines of code
  • tests: 27 files and 7,433 lines of code
  • pkg: 26 files and 4,731 lines of code
  • types: 20 files and 3,522 lines of code
  • protocol: 9 files and 1,095 lines of code
  • root: 5 files and 639 lines of code

Languages

By file count: Go 237 files, Java 33 files, Shell 5 files, Dockerfile 2 files, Makefile 1 file.

Documentation

Repowise documents olake in 460 pages generated from the source, 27 of them written explanations of the repository and its modules. Module guides include:

  • drivers/abstract: Abstract Driver Framework
  • pkg/binlog: Binlog Reading Utilities
  • root: Connector Entry Point

More on olake: summary and FAQ, codebase overview, code health.

Related guide: Best dependency graph tools for monorepos