Skip to main content

    AI risk and compliance

    Algorithmic bias in business AI

    Algorithmic bias comes from historical data, feature choices, model design and deployment in the wrong context. This article helps Australian organisations understand where bias enters business AI and how to detect, measure and reduce it. Bias management needs monitoring from development through to operation, because biased systems create legal liability, reputational damage and operational failure.

    In short

    Bias is a legal risk under Australian anti-discrimination law

    • Algorithmic bias is a legal risk under Australian anti-discrimination law. Liability can arise whether a person or an algorithm made the decision.
    • Bias enters through training data that reflects past discrimination and through models used outside their intended context. It also enters through proxy variables, which stand in for protected attributes such as race, sex or age.
    • High-risk AI, such as hiring, credit, insurance and pricing systems, should be tested for bias before deployment and audited regularly after.
    • From 10 December 2026, Australian Privacy Principle (APP) 1.7 and APP 1.8 set privacy policy disclosure rules for some automated decisions. They set disclosure rules, and anti-discrimination law sets the fairness standard.

    Detail

    Bias sources include historical data and proxy variables

    If your AI was trained on your historical hiring data, it learned your organisation's past biases. If your credit model uses postcode as a feature, it may be using a proxy for ethnicity. If your insurance pricing AI optimises for profit without fairness constraints, it may disadvantage groups that happen to be less profitable. None of this needs to be intended for the organisation to be liable.

    Algorithmic bias occurs when AI systems produce systematically unfair outcomes for particular groups. Its sources are well understood. Training data bias comes from historical data that reflects past discrimination, under-represents some groups or varies in quality across groups. Feature selection bias comes from variables linked to protected attributes, even when those attributes are left out of the model. Postcode can predict ethnicity, a name can predict gender and an employment gap can predict caring responsibilities.

    Model design bias comes from objectives that disadvantage groups without anyone intending it. Training a model only for accuracy on past outcomes builds past unfairness into it. Deployment context bias comes from using a model built on data about one group of people with a different group.

    The high-risk domains are hiring and recruitment, credit and lending, insurance pricing and underwriting, and customer pricing and promotions. Fraud detection is another, because it can flag some groups of customers more often than others.

    Mitigation takes active effort. It needs diverse training data and model design that takes fairness into account. It also needs monitoring of protected attributes and human oversight of consequential decisions. Regular independent bias audits and a clear way for people to challenge AI decisions complete the set.

    Australian anti-discrimination law already applies to automated decisions. The Sex Discrimination Act 1984, Racial Discrimination Act 1975, Age Discrimination Act 2004 and Disability Discrimination Act 1992 have no exception for algorithms. If your AI produces a discriminatory outcome, the organisation can be liable even though no person intended it.

    From 10 December 2026, APP 1.7 and APP 1.8 set privacy policy disclosure rules for some automated decisions. Our article on what APP 1.7 requires sets out the test in full. These are disclosure rules, and they set no fairness test.

    Commercial impact

    Bias brings legal, commercial and reputational costs

    Australian anti-discrimination law applies to automated decisions, and the 4 main federal Acts contain no exception for algorithms. If your AI produces discriminatory outcomes, your organisation can be liable.

    Bias also costs money. Biased customer segmentation misses revenue from under-served groups. Biased hiring narrows your talent pool. Biased fraud detection creates friction for legitimate customers who happen to match a profile. Each of these is a business cost that looks like an optimisation.

    Bias also carries reputational risk. Bias cases overseas have led to legal action, regulatory scrutiny and lasting damage to brands. Australian regulators and media also follow these issues closely.

    Bias stays invisible until someone tests for it. Leaders often assume their AI is fair because nobody designed it to discriminate. Testing shows whether that assumption holds.

    For boards, this connects to Report 798 from the Australian Securities and Investments Commission (ASIC). It states that directors' and officers' duties of reasonable care and diligence 'extend to the adoption, deployment and use of AI'. A board that allows high-impact AI to be deployed without bias testing or oversight should expect questions about how it met those duties.

    In practice

    Illustrative cases of bias in hiring, credit and pricing AI

    Illustrative composite: a recruitment platform's AI screening tool showed significant gender bias for technical roles. The training data came from years of hiring at companies with mostly male engineering teams, so the AI learned to favour male candidates. The remedy is to add fairness rules to the model, rebalance the training data across groups and check results against hiring quality measures.

    Illustrative composite: a lender's credit AI rejected younger applicants more often than their creditworthiness justified. The model used years at the current address and length of credit history as features, and both are proxies for age. The remedy is to replace them with direct measures of creditworthiness.

    Illustrative composite: a retailer's AI pricing engine offered different prices by postcode. When the team mapped the prices against demographic data, they correlated clearly with ethnicity. The pricing team had not set out to discriminate. It had optimised for willingness to pay, which in its data correlated with socioeconomic factors tied to postcode. An ethics review flagged the issue before deployment, and fixing it meant redesigning the model's objectives.

    Illustrative composite: an insurer deployed risk assessment AI without bias testing. A pattern of complaints prompted analysis after deployment. The analysis showed the model charged higher premiums in particular areas, in ways that correlated with ethnicity.

    What to do

    Steps to test for and reduce bias in AI systems

    • Require bias testing before deployment for any AI that affects individuals, including hiring, credit, insurance, pricing and fraud detection.
    • Audit training data for historical bias, under-representation and proxy variables that correlate with protected attributes.
    • Make fairness part of what each model aims for, as well as accuracy on historical outcomes.
    • Provide human oversight and a way for individuals to challenge every consequential AI decision.
    • Schedule independent bias audits each year for deployed systems, and document all testing so it is ready if a regulator asks.

    Last reviewed

    Put this topic against your own numbers, owners and timelines.