salem-import/README.md

# Salem Oregon Address Import

See [https://wiki.openstreetmap.org/wiki/Salem_Oregon_Address_Import](https://wiki.openstreetmap.org/wiki/Salem_Oregon_Address_Import)

## Data Source

- Salem GIS department's FTP server
  - Also https://data.cityofsalem.net/datasets/salem::primary-address/explore
- No copyright, public domain

<img src="Screenshot_2023-10-25_213128.png"/>

## Field Mapping

- ADD_NUM -> addr:housenumber (int)
- formatstreet(FULL_ST_NAME) -> addr:street (text)
- SUB_VAL -> addr:unit (text)
- title(CITY) -> addr:city (text)
- ZIP -> addr:postcode (text)
- addr:state 'OR' (manually added)

## Processing

- Get `Primary_Addresses.geojson` from the City GIS server linked above.
- Open the file in QGIS and open the layer attribute table
- Use the field calculator and the below script to add new virtual fields per the mapping above. (You may have to hit Save and Load Functions again to refresh QGIS's memory)
- Export the layer to geojson with "EPSG:4326 - WGS 84" projection. Deselect all fields besides those above.
- Save as `processed.geojson`

## Import steps

- Login to JOSM as your import account user (you can see this in the title bar)
- Ensure that the MapWithAI plugin is installed and activated (for its validators)
- Under Preferences > Data Validator, turn on Include POIs under Addresses. (Using my custom version of JOSM)
- Load processed.geojson into JOSM
- For each area you wish to work on, repeat:
  - Zoom in to the area you wish to work on and Download OSM data to a New Layer
  - In the geojson layer, select one neighborhood or city block worth of addresses
  - Click Edit > **Merge Selection** (it will warn that this isn't the best, continue since you have approval for this import and are taking precautions.)
    - Make sure you don't accidentally try and upload the processed.geojson data layer directly to OpenStreetMap! The Continue warning looks similar.
  - **Delete the selected items you just merged (so you don't try merging them again later)**
  - Switch to Data Layer 1 and run the JOSM Validator
    - An easy way to only validate changes is to press the Upload button, but cancel before actually uploading.
    - We don't need to worry about validation errors that don't involve our changes.
  - Fix all duplicate housenumber warnings and nearby street not found warnings
    - An easy way to auto-fix all duplicate housenumbers is to **select all duplicates, Find In Selection for `new`, and delete.**
    - Pay special attention to errors like "East Street Northeast" -> "E Street Northeast" which can be mass-corrected.
    - Skim the results to see if there are common errors like existing addresses misspelled with new correct addresses over them.
  - **Search for all `new "addr:housenumber" = "0"` elements and delete them.**
  - Click Upload, verify there are no further warnings or errors in the changeset
    - Make sure there are no erroneous Relations or other unwanted objects about to be uploaded.
    - The Upload window should ideally only indicate that you're adding new objects, not modifying or deleting anything (unless of course you intend to.)
  - **Upload with this changeset comment** (use the triple "+" icon to paste under Settings):

```
comment=Addresses near Salem Oregon #salemimport
import=yes
website=https://wiki.openstreetmap.org/wiki/Salem_Oregon_Address_Import
source=City of Salem GIS
source:url=https://data.cityofsalem.net/datasets/salem::primary-address/explore
```

  - Review imported data in Achavi or Osmcha to ensure it looks proper.

## QGIS Processing script

```
import qgis.core
import qgis.gui
import re

#
# This will keep street names like SR 574A as SR 574A however
# will lowercase other number-digit suffixes with <2 or >4 numbers
# or >1 suffix-letters, like 12th Street or 243rd Ave.
#

@qgsfunction(args='auto', group='Custom', referenced_columns=[])
def getstreetfromaddress(value1, feature, parent):
    parts = value1.split()
    parts.pop(0) # Ignore the first bit (i.e. "123" in "123 N MAIN ST")
    parts = map(formatstreetname, parts)
    return " ".join(parts)

@qgsfunction(args='auto', group='Custom', referenced_columns=[])
def formatstreet(value1, feature, parent):
    parts = value1.split()
    # Handle the special case of a street name starting with "ST"
    # which is almost always "Saint __" and not "Street __"
    if parts[0] == "ST":
        parts[0] = "Saint"
    parts = map(formatstreetname, parts)
    return " ".join(parts)

# Internal function
def formatstreetname(name):
    # Acryonyms
    if name == "CR":
        return "County Road"
    if name == "SR":
        return "SR" # State Route
    if name == "NFS":
        return "NFS" # National Forest Service?
    if name == "US":
        return "US"
    # Directions
    if name == "N":
        return "North"
    if name == "NE":
        return "Northeast"
    if name == "E":
        return "East"
    if name == "SE":
        return "Southeast"
    if name == "S":
        return "South"
    if name == "SW":
        return "Southwest"
    if name == "W":
        return "West"
    if name == "NW":
        return "Northwest"
    # Suffixes
    if name == "AV":
        return "Avenue"
    if name == "AVE":
        return "Avenue"
    if name == "BLVD":
        return "Boulevard"
    if name == "BND":
        return "Bend"
    if name == "CIR":
        return "Circle"
    if name == "CR":
        return "Circle"
    if name == "CT":
        return "Court"
    if name == "DR":
        return "Drive"
    if name == "FLDS":
        return "Fields"
    if name == "GRV":
        return "Grove"
    if name == "HL":
        return "Hill"
    if name == "HOLW":
        return "Hollow"
    if name == "HW":
        return "Highway"
    if name == "HWY":
        return "Highway"
    if name == "LN":
        return "Lane"
    if name == "LOOP":
        return "Loop"
    if name == "LP":
        return "Loop"
    if name == "PATH":
        return "Path"
    if name == "PL":
        return "Place"
    if name == "RD":
        return "Road"
    if name == "RUN":
        return "Run"
    if name == "SQ":
        return "Square"
    if name == "ST":
        return "Street"
    if name == "TER":
        return "Terrace"
    if name == "TRL":
        return "Trail"
    if name == "VW":
        return "View"
    if name == "WAY":
        return "Way"
    if name == "WY":
        return "Way"
    if name == "XING":
        return "Crossing"
    if re.match('^[0-9]{2,4}[A-Za-z]$', name) != None:
        return name

    return name.capitalize()
```