osm-import-tools/README.md

# OSM Import Tools

### Main Dashboard
![Main Dashboard](screenshots/download.png)

### Map Viewer
![Map Viewer](screenshots/map.png)

## Using

For each county, run the four steps in order using the dashboard. Each step must complete before the next:

1. **Download county data** — fetches the county dataset (or upload a shapefile ZIP for paths)
2. **Download OSM data** — queries Overpass for the relevant features in the county
3. **Convert to GeoJSON** — converts the county data to OSM-tagged GeoJSON
4. **Diff vs OSM** — compares county data against OSM; produces added (green) and removed (red) GeoJSON

Open the result in the Map Viewer. Green = features in county data missing from OSM (candidates to add). Red = features in OSM not in county data. Hide the OSM (gray) and county (purple) layers to focus on the diff. Click a feature to inspect its properties and accept or exclude it. Open accepted features in JOSM for final review before uploading.

### Roads

Queries Overpass for all highway types: motorway, trunk, primary, secondary, tertiary, residential, unclassified, and their link variants, plus service and track.

### Addresses

Run roads first — having street geometries in JOSM reduces address validation errors during upload. Queries Overpass for all `addr:*` nodes and ways. Easiest to also hide removed (red) and focus only on added (green) addresses missing from OSM.

### Paths

Only Sumter currently has paths available. Queries Overpass for foot, bicycle, and multi-use paths.

## Docker Quick Start

### Using Docker Compose (Recommended)

1. Build and start the container:
```bash
docker-compose up -d --build
```

Note: The `--build` flag ensures the image is rebuilt with the latest code changes.

2. Access the web interface:
- Main interface: http://localhost:5000
- Map viewer: http://localhost:5000/map

3. Stop the container:
```bash
docker-compose down
```

### Using Docker Directly

1. Build the image:
```bash
docker build -t osm-import-tools .
```

2. Run the container:
```bash
docker run -d \
  -p 5000:5000 \
  -v "$(pwd)/data:/data" \
  --name osm-import-tools \
  osm-import-tools
```

3. View logs:
```bash
docker logs -f osm-import-tools
```

4. Stop the container:
```bash
docker stop osm-import-tools
docker rm osm-import-tools
```

### Deploying with Portainer

Just push to git and set up a pull from git; Portainer can build and run the project directly.

### Uploading to a registry
```bash
docker build -t osm-import-tools .
docker tag osm-import-tools your-registry.example.com/you/osm-import-tools
docker push your-registry.example.com/you/osm-import-tools
```

## Features

### Main Dashboard (/)
- Run data processing scripts per county (Roads, Paths, Addresses)
- View real-time script output with colored status indicators
- Live CPU/memory/disk usage charts during execution
- Cancel running jobs
- Quick-access links to processed GeoJSON files

### Map Viewer (/map)
- Interactive Leaflet map for comparing GeoJSON layers
- Layer controls with drag-and-drop reordering
- Filter by added (green) / removed (red) features
- Hide service, track, and unclassified roads
- Click features to inspect properties
- Accept/reject diff features and save results
- Multi-select mode (or Shift+drag) for bulk operations

### Counties (/counties)
- Define counties with ID, name, state, color, and download URLs
- Download/upload the `counties.yml` config file

### Street Exceptions (/exceptions)
- Manage street name corrections (e.g. "D Angelo" → "D'Angelo")
- Download/upload the `exceptions.yml` config file

### Exclusions (/exclusions)
- Define field/value pairs to exclude from diff output
- Exclude button on map popups for quick additions
- Download/upload the `exclusions.yml` config file

### Authentication
- Login required for all write operations
- Admin user configured via `ADMIN_PASSWORD` environment variable

## Volume Mounts

The Docker container mounts a single data directory:
- `./data` → `/data` - All data files, organized by county

Inside `/data`, the structure is:
- `/data/lake/` - Lake County data
- `/data/sumter/` - Sumter County data
- `/data/counties.yml` - County definitions
- `/data/exceptions.yml` - Street name corrections
- `/data/exclusions.yml` - Exclusion rules
- `/data/users.yml` - User credentials

All changes are persisted on the host in the local `./data` folder.

## API Endpoints

### Auth
- `GET /api/auth-status` - Check authentication status

### Script Execution
- `POST /api/run-script` - Execute a processing script
- `GET /api/job-status/<job_id>` - Get job status and logs
- `POST /api/cancel-job` - Cancel a running job

### File Operations
- `GET /api/list-files` - List available GeoJSON files
- `POST /api/upload-file` - Upload shapefile ZIP
- `POST /api/clear-data` - Delete all county data files
- `GET /data/<path>` - Serve data files

### Configuration
- `GET /api/config/<filename>` - Download a YAML config file
- `POST /api/config/<filename>` - Upload a YAML config file
- `GET/POST /api/counties` - Get or save county configuration
- `GET/POST /api/exceptions` - Get or save street exceptions
- `GET/POST /api/exclusions` - Get or save exclusions
- `POST /api/exclusions/add` - Append new exclusions (skips duplicates)

### System
- `GET /api/system-stats` - CPU, memory, and disk usage

## Troubleshooting

### Port already in use
If port 5000 is already in use, edit `docker-compose.yml`:
```yaml
ports:
  - "8080:5000"  # Change 8080 to any available port
```

### Permission issues
Ensure the data directory has proper permissions:
```bash
mkdir -p data
chmod -R 755 data
```

### View container logs
```bash
docker-compose logs -f
```

### Rebuild after code changes
```bash
docker-compose down
docker-compose build --no-cache
docker-compose up -d
```

## TODO

- Validate whether a similar address is within 50 feet (same number?)
- If addr:unit is like "Apartment 123" or "Apartment 123;Apartment 124;Apartment 125" remove the word Apartment so it's just "123" or "123;124;125"
- If an item gets through processing and is "<Null>" it should be cleared
- If a value is a range like "123 - 456" assume it's the same as "123-456" for matching purposes
- Unit numbers with semicolons are still getting detected as different despite being identical, like:
addr:city=Wildwood
addr:housenumber=8655
addr:postcode=34785
addr:state=FL
addr:street=Easterly Lane
addr:unit=108;208;308
- Some items appear duplicate in the data itself for unknown reasons:
addr:city=Bushnell
addr:housenumber=5740
addr:postcode=33513
addr:state=FL
addr:street=Hideaway Circle
- Sr is SR (State Route)