We present a pixel-level angle sensitive detector composed of silicon-on-insulator (SOI) photodiode (PD) stacked with a gold surface plasmon (SP) antenna to affect the direction of the incoming light. The surface plasmons are excited in the grating-type SP antenna and enhance the diffraction efficiency of the grating. The diffracted light is coupled strongly with the propagation light in the SOI waveguide when the phase matching condition is satisfied. The phase matching takes place at a specific angle of light incidence, and the discrimination of the light based on the incident angle is achieved. As spatial patterns in the polar coordinate of the elevation-azimuth angles (θ, ϕ) of the incident light, we present the phase matching condition theoretically, the absorption efficiency in the SOI by simulation, and also the quantum efficiency of the SOI PD experimentally for different SP antennas of one-dimensional (1D) line-and-space (L/S) and two-dimensional (2D) hole array gratings under various polarization angles. 1D grating offers a polarization sensitive angle detection and 2D grating exhibits angle detection in two orthogonal directions, enabling a polarization independent angle sensitivity. A good agreement among the theory, simulation, and experiment are attained. The proposed device features relatively high quantum efficiency as an angle-sensitive pixel (ASP) and gives wider opportunities in applications such as three-dimensional (3D) imaging, depth-of-field extension, and lensless imaging.