Converting 2D technical drawings into 3D models is one of the most persistent challenges in engineering, architecture and manufacturing. A civil engineer has thousands of orthographic floor plans. A manufacturer has a library of 2D drawings created before 3D CAD was standard. A game studio has concept art that needs to become geometry. In each case the question is the same: can we automate the conversion from 2D into usable 3D geometry, and if so, how?
This article documents our research into procedural approaches to 3D model generation from 2D orthographic views — combining classical computer vision, geometric reconstruction algorithms, and modern deep learning. This is active research territory with significant unsolved problems, and this article is honest about both what works and what remains hard.
Three Distinct Problem Formulations
There are three meaningfully different problems: (1) orthographic view reconstruction — front, top and side engineering drawings to reconstruct a 3D solid; (2) single-image 3D inference — one photograph to infer 3D structure; (3) sketch-to-3D — a hand-drawn concept sketch to generate a plausible 3D model. Our research focuses on orthographic view reconstruction because it has direct application in engineering contexts where geometric accuracy matters.
Classical Approach: CSG Reconstruction
The classical approach uses Constructive Solid Geometry (CSG) operations. The algorithm: extract silhouettes from the front, top and side views; extrude each silhouette orthogonally into a 3D solid; compute the intersection of the three solids. The result is the set of all 3D points consistent with all three views. This works well for objects whose geometry is captured accurately by three orthogonal silhouettes — many machined parts, furniture items and architectural elements fall into this category.
We implemented this using OpenCV for silhouette extraction and Open3D for CSG operations. For clean engineering drawings with clearly defined line weights, reconstruction quality is sufficient for further refinement. For complex drawings with many overlapping lines and annotations, the silhouette extraction step requires significant preprocessing to isolate geometry from annotation.
Deep Learning Approaches
Transformer-Based 3D Generation
Large-scale transformer models trained on paired 2D/3D datasets are the most exciting recent development in this space. Models like Shap-E and TripoSR can generate 3D models from single images with good qualitative results for common object categories. For engineering drawings, the main limitation is distribution shift — models trained on photographs of consumer objects do not generalise well to technical line drawings of custom-engineered parts without domain-specific fine-tuning.
NeRF and Sparse-View Methods
Neural Radiance Fields and related implicit representations learn 3D geometry from multiple images with known camera poses. Engineering drawings provide only three views with parallel projection, not the perspective projections these methods assume. Recent sparse-view variants show promise but accuracy for precise engineering geometry remains limited without dense multi-view input.
Our Hybrid Pipeline
Our best-performing approach combines classical and learned methods. The pipeline: preprocess input drawings to extract clean silhouettes using a trained segmentation model; apply CSG intersection to get a coarse 3D solid consistent with all three views; use a fine-tuned PointNet++ to predict surface geometry in ambiguous regions; apply Poisson surface reconstruction to produce a clean mesh; post-process to remove floating geometry and fill holes. This hybrid produces geometrically accurate results for object categories well-represented in the training data.
Practical Applications
Architectural Floorplan to 3D
Architectural floor plans have strong domain constraints — rooms are bounded polygonal regions, walls have standard thicknesses, doors and windows appear as standard symbols. We built a floorplan-to-3D system using a fine-tuned YOLOv8 model to detect rooms, walls, doors and windows, followed by a rule-based 3D extrusion system. The result is a navigable 3D model suitable for architectural visualisation, VR walkthroughs and spatial analysis.
Legacy CAD Library Conversion
Manufacturing companies often have large libraries of legacy 2D CAD drawings (DXF, DWG) created before 3D CAD was standard. For DXF-format drawings, we parse vector geometry directly using the ezdxf Python library, extracting precise silhouettes without image-based noise. This significantly improves CSG reconstruction quality compared to working from scanned images.
Open Problems
Several challenges remain. Internal structure (hollow objects, threaded holes, internal channels) cannot be inferred from external orthographic views alone — cross-section views or domain knowledge is required. Material and texture assignment requires a separate classification model or user input. Assembly relationships between components are not recoverable from individual part drawings. The most promising future direction is combining large-scale 3D foundation models with domain-specific fine-tuning and classical geometric constraints.
Key Takeaways
- Distinguish the three 2D-to-3D problem types — they require fundamentally different approaches.
- Classical CSG intersection is a strong baseline for orthographic reconstruction of geometrically simple objects.
- Transformer-based models require domain-specific fine-tuning to generalise to engineering drawings.
- A hybrid pipeline combining classical geometry with learned ambiguity resolution outperforms either approach alone.
- For DXF/DWG format drawings, parse vector geometry directly — image-based silhouette extraction adds unnecessary noise.
- Internal structure and assembly relationships remain open problems requiring additional information beyond three views.
