LEGO Connect Language (LCL): A Textual Approach to LEGO Assembly

2023

Overview

Building LEGO sets relies on visual instructions. The LEGO Connect Language (LCL) is here proposed as a possible benchmark for LLMs and as an alternative to complex visual inputs by providing a structured, textual approach to LEGO assembly, making the process accessible to a LLM and verifiable through a systematic method.

Understanding LCL Syntax

LCL instructions guide the assembly process. For example, "S1: PID: 3001, O: N-U, CP: [(1,1), (1,2)], A: Attach" instructs to attach a piece with PID 3001, oriented north-up, at position of the first two holes as the first step.

LCL Syntax and Semantics

LCL comprises several elements:

  1. Piece Identification (PID): An alphanumeric code for each LEGO piece.
  2. Orientation (O): Denotes the piece's orientation using compass directions and vertical alignment (based on an initial arbitrary definition of N/S of the piece).
  3. Connection Point (CP): Grid coordinates on the base piece for attachment based on N/S.
  4. Action (A): The action to be performed, like attaching or removing.
  5. Sequence (S): A numerical order of construction steps.

Encoding Process

An encoding algorithm could then be devised which transforms visual instructions into LCL format, involving piece recognition, orientation detection, connection mapping, and sequential ordering.

  1. Define a standard reference orientation for each type of LEGO piece, based on its most distinctive or common feature.
  2. Map the standard reference orientation to compass directions. For instance, the longest side of a piece could be defined as facing North in its standard orientation.
  3. Establish rules for rotating the piece from the standard orientation. Detail the resulting orientation for each type of rotation (e.g., 90 degrees clockwise).
  4. Include the standard orientation and rotation rules in the LCL documentation for user reference.

Decoding and Verification

As well as a decoder algorithm which could validate a built model against LCL instructions, involving step-by-step comparison and mismatch identification.

Formalization

We define a LEGO piece in LCL as \( L = (PID, O, CP, A, S) \). Given LCL instructions \( \mathcal{I} = \{L_1, L_2, ..., L_n\} \) and a user's build sequence \( \mathcal{U} = \{U_1, U_2, ..., U_m\} \), the decoder function \( D(\mathcal{I}, \mathcal{U}) \) identifies mismatches \( \mathcal{M} \): \[ D(\mathcal{I}, \mathcal{U}) = \{i \mid L_i \in \mathcal{I}, U_i \in \mathcal{U}, L_i \neq U_i \} \] The objective is minimizing \( |\mathcal{M}| \), where an empty set signifies a perfect match.

Example

  • {'PID': '3040', 'O': 'N-U', 'CP': ['(1,1)'], 'A': 'Attach', 'S': 1},
  • {'PID': '3039', 'O': 'S-U', 'CP': ['(1,2)', '(1,3)'], 'A': 'Attach', 'S': 2},
  • {'PID': '3660', 'O': 'E-D', 'CP': ['(2,1)', '(2,2)'], 'A': 'Attach', 'S': 3},
  • {'PID': '3002', 'O': 'W-U', 'CP': ['(2,3)'], 'A': 'Attach', 'S': 4},
  • {'PID': '3022', 'O': 'N-U', 'CP': ['(1,4)'], 'A': 'Attach', 'S': 5},
  • {'PID': '3004', 'O': 'S-U', 'CP': ['(2,4)'], 'A': 'Attach', 'S': 6},

Simple Dataset

Apple

Construct a LEGO apple with a mix of red and green colors, resembling a typical apple shape using slopes and bricks.

  • Green Slope 45 2 x 1 - Code: 3040
  • Red Slope 45 2 x 2 - Code: 3039 (Quantity: 2)
  • Lime Slope, Inverted 45 2 x 2 - Code: 3660 (Quantity: 2)
  • Red Brick 2 x 3 - Code: 3002
  • Lime Plate 2 x 2 - Code: 3022
  • Lime Brick 1 x 2 - Code: 3004

Yellow Hut

Build a sturdy hut with a purple and yellow color scheme, featuring a simple structure and a sloped roof.

  • Trans-Clear Brick 1 x 2 without Bottom Tube - Code: 3065 (Quantity: 2)
  • Medium Nougat Brick 2 x 2 - Code: 3003
  • Lime Plate 2 x 6 - Code: 3795
  • Bright Light Yellow Brick 1 x 2 - Code: 3004 (Quantity: 4)
  • Bright Light Yellow Brick 2 x 2 - Code: 3003
  • Medium Lavender Slope 45 2 x 2 - Code: 3039 (Quantity: 4)

Fortress

Create a medieval-themed LEGO fortress with arches, walls, and defensive structures, symbolizing a stronghold.

  • Green Plate 2 x 8 - Code: 3034
  • Light Bluish Gray Arch 1 x 4 x 2 - Code: 6182 (Quantity: 2)
  • Sand Green Brick 1 x 2 - Code: 3004 (Quantity: 2)
  • Light Bluish Gray Brick 1 x 2 - Code: 3004 (Quantity: 2)
  • Dark Bluish Gray Brick 1 x 2 - Code: 3004 (Quantity: 2)
  • Light Bluish Gray Brick 2 x 2 - Code: 3003
  • Dark Purple Slope 45 2 x 1 - Code: 3040 (Quantity: 2)
  • Reddish Brown Brick, Round 1 x 1 Open Stud - Code: 3062b (Quantity: 2)

Dinghy

Assemble a small LEGO dinghy with a white sail and a sturdy mast, designed for ocean adventures.

  • Dark Tan Plate 2 x 4 - Code: 3020
  • Tan Slope, Inverted 33 3 x 2 with Flat Bottom Pin and Connections - Code: 3747b
  • White Slope 45 2 x 2 - Code: 3039 (Quantity: 3)
  • White Brick 2 x 2 - Code: 3003
  • White Brick 1 x 2 - Code: 3004
  • Tan Brick 2 x 3 - Code: 3002
  • Reddish Brown Brick, Round 2 x 2 with Axle Hole - Code: 3941

Blue Bot

Construct a friendly-looking LEGO robot with a humanoid structure, featuring a distinguishable head, body, arms, and legs.

  • Medium Blue Brick 2 x 2 - Code: 3003
  • Brick, Modified 2 x 3 with Curved Top - Code: 6215
  • Brick 2 x 4 - Code: 3001
  • Brick 1 x 2 - Code: 3004 (Quantity: 2)
  • Brick, Round 2 x 2 with Grille - Code: 92947
  • Plate 2 x 2 - Code: 3022
  • Brick, Modified 1 x 2 with Studs on 1 Side - Code: 11211
  • Brick 1 x 2 without Bottom Tube - Code: 3065
  • Tile 1 x 1 Round - Code: 98138
  • Brick, Round 2 x 2 Dome Top, with Bottom Axle Holder - Code: 553c

Toy Car

Build a zippy, colorful LEGO toy car with a compact design, featuring wheels, a sloped windshield, and a streamlined look.

  • Brick 2 x 6 - Code: 2456
  • Slope 2 x 2 45° - Code: 3039
  • Brick 1 x 2 without Bottom Tube - Code: 3065
  • Brick 1 x 2 - Code: 3004
  • Plate 2 x 2 with Wheel Holders - Code: 4600 (Quantity: 2)
  • Wheel 8mm D. x 6mm with Slot - Code: 34337 (Quantity: 4)
  • Tire Offset Tread Small - Band Around Center of Tread - Code: 87414 (Quantity: 4)

Goldfish

Create a LEGO goldfish with fins and tail, featuring a colorful design and elements for eyes and detailed appearance.

  • Brick 2 x 4 - Code: 3001 (Quantity: 2)
  • Brick 1 x 2 with Pin Hole - Code: 3700
  • Brick, Modified 1 x 2 with Studs on 1 Side - Code: 11211 (Quantity: 2)
  • Brick 2 x 3 - Code: 3002
  • Slope 45° 2 x 2 - Inverted - Code: 3660
  • Slope 2 x 1 - 45° - Code: 3040 (Quantity: 4)
  • Tile 1 x 1 Round with Eye Pattern - Code: 98138pb007 (Quantity: 2)
  • Slope 30° 1 x 2 x 2/3 - Code: 85984

Baby Elephant

Assemble a LEGO baby elephant with a focus on its trunk, ears, and body structure, featuring a friendly and realistic appearance.

  • Brick 2 x 6 - Code: 2456
  • Brick 1 x 2 - Code: 3004 (Quantity: 3)
  • Brick 1 x 4 - Code: 3010
  • Brick 1 x 1 with Stud on 1 Side - Code: 87087 (Quantity: 2)
  • Tile 1 x 1 Round with Eye Pattern - Code: 98138pb027 (Quantity: 2)
  • Brick 2 x 4 - Code: 3001

Flamingo

Construct a LEGO flamingo with pink bricks, designed to stand on one leg and feature a long neck and beak.

  • Brick 1 x 2 - Code: 3004 (Quantity: 3)
  • Brick, Modified 2 x 3 with Curved Top - Code: 6215 (Quantity: 2)
  • Brick 1 x 1 with Stud on 1 Side - Code: 87087 (Quantity: 2)
  • Plate 2 x 3 - Code: 3021
  • Slope 2 x 2- 45° - Code: 3039
  • Tile 1 x 1 Round with Eye Closed Pattern - Code: 98138pb028 (Quantity: 2)

Twin Engine Airplane

Build a LEGO twin-engine airplane, with a streamlined body, wings, and a tail, suitable for various exploratory missions.

  • Plate 2 x 8 - Code: 3034 (Quantity: 2)
  • Brick 1 x 2 x 2 with Inside Stud Holder - Code: 3245c
  • Brick, Modified 1 x 1 x 1 2/3 with Studs on 1 Side - Code: 32952 (Quantity: 2)
  • Brick 1 x 4 with 4 Studs on 1 Side - Code: 30414 (Quantity: 2)
  • Slope 2 x 2- 45° - Code: 3039
  • Brick 1 x 2 without Bottom Tube - Code: 3065

Preprint

Show, Don't Tell: Evaluating Large Language Models Beyond Textual Understanding with ChildPlay
Gonçalo Hora de Carvalho, Oscar Knap, Robert Pollice; 2024

Read Preprint