Running AI Behind the Firewall: What Construction Finance Teams Need to Know
As interest in AI continues to grow across the construction industry, one big question keeps coming up: how can we use large language models (LLMs) like GPT-4 without compromising our confidential project data?
At CFMA, we’ve seen an increasing number of finance and IT professionals asking about the trade-offs between cloud-based AI services and running AI tools locally or “behind the firewall.” In this blog post, we’ll explore what it really takes to run AI securely on-premise, break down the key hardware and software involved, and introduce a practical starting point for construction teams: Ollama, an open-source solution for local AI.
Understanding AI Inference Hardware
To run an AI model like GPT-4, you’re not just installing software—you’re activating incredibly complex systems that require high-end hardware known as AI inference hardware.
Unlike training (which teaches an AI model how to generate results), inference is what happens when a model processes your input and produces an output in real-time.
Common inference hardware includes:
- GPUs (Graphics Processing Units) – e.g., NVIDIA A100, H100, or L40S.
- ASICs (Application-Specific Integrated Circuits) – like Google’s TPUs.
- FPGAs (Field-Programmable Gate Arrays) – used for niche AI workloads.
- Edge AI Chips – built for small-scale models on devices like IoT hardware.
LLMs such as GPT-4 typically run on powerful data center GPUs like the NVIDIA A100 or H100. These aren’t cheap—just one A100 80GB GPU costs $25,000–$35,000, and high-throughput workloads often require dozens or even hundreds of them.
Why Running AI On-Premises Is So Challenging
While the idea of keeping your AI inside your network sounds appealing for data security reasons, here are the main barriers construction companies face:
- Hardware cost: Total investment easily runs into the millions when accounting for servers, cooling systems, and racks.
- Infrastructure demands: You’ll need a data center with robust power, cooling, and physical security.
- Technical complexity: Specialized IT staff must manage deployments, orchestration tools, and GPU drivers.
For smaller open-source models (7B–13B parameters), on-prem AI is possible. But running something like GPT-4 locally is currently limited to large cloud providers or research labs.
Why the Cloud Is Still Common — and What the Risks Are
Most companies rely on cloud-based AI providers like OpenAI, Anthropic, or Cohere because:
- No need for up-front hardware investment
- Scalable infrastructure
- Managed software environments and APIs
However, using cloud AI requires trust in your provider’s data governance policies. Even if you opt out of training, there’s always a residual risk that data could be mishandled, or worse, exposed via prompt-based “jailbreaks.”
For example, instruction-tuned models like GPT can sometimes leak sensitive information if not properly secured. That’s why strong opt-out policies, internal controls, and audit mechanisms are crucial.
A More Practical Path: Running Smaller AI Models Locally with Ollama
For teams that want to test AI securely behind their firewall, we recommend starting with Ollama — a free, open-source tool that simplifies running open-source LLMs (like Llama 2, Mistral, and Gemma) locally.
What Is Ollama?
Ollama is a lightweight framework that helps you:
- Download and run open-source LLMs directly on your machine
- Manage multiple models
- Interact with AI through both command-line and user-friendly GUIs (like Open WebUI)
It’s ideal for teams that want to:
- Keep sensitive data in-house
- Avoid recurring vendor fees
- Customize or fine-tune models for internal use cases
Key Limitations to Keep in Mind:
- Model size: Smaller than GPT-4, so performance and fluency may differ.
- Hardware requirements: Mid- to high-end GPUs still needed, though much less costly than GPT-scale clusters.
- Context limits: Local models often can’t handle as much data at once.
- Domain tuning: You may need to experiment with prompts or fine-tuning for construction-specific workflows.
Upcoming Training: Learn to Use Ollama for Construction AI
We’re planning a free training session for CFMA members on how to get started with Ollama. No need to install anything during the session — we’ll walk through:
- Installing and configuring Ollama
- Hardware and security considerations
- Realistic expectations for local AI
- Practical use cases in forecasting, scheduling, and reporting
If you’re interested in attending or want your team to join, let us know. This is a great opportunity for construction IT and finance professionals to explore AI without risking sensitive data exposure.
TL;DR: Your AI Deployment Options
✅ Cloud AI (like GPT-4)
Great for ease of use and performance, but requires strict data governance.
✅ On-Prem with Ollama + Open-Source Models
A practical way to keep data local while testing real-world use cases.
✅ Hybrid approach
Use secure cloud tools for general use, and run sensitive workloads in-house with smaller models.
Let’s continue the conversation: How are you approaching AI in your company? Are you experimenting with open-source models or sticking with the cloud for now?
We’d love to hear your perspective — and help you get started with secure, effective AI adoption.
